----------
OS WAIT ARRAY INFO: reservation count 77805, signal count 105351
--Thread 139679652914944 has waited at dict0dict.c line 742 for 258.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
--Thread 139679655577344 has waited at buf0buf.c line 2527 for 257.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
--Thread 139679656908544 has waited at buf0buf.c line 2527 for 257.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
--Thread 139679656642304 has waited at buf0buf.c line 2527 for 257.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
--Thread 139679654778624 has waited at buf0buf.c line 2527 for 257.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
--Thread 139656811095808 has waited at buf0flu.c line 1481 for 256.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
--Thread 139679652382464 has waited at buf0buf.c line 2527 for 256.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
--Thread 139679661434624 has waited at buf0buf.c line 2527 for 256.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
--Thread 139679663032064 has waited at buf0buf.c line 2527 for 256.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
--Thread 139679665428224 has waited at dict0dict.c line 742 for 253.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
--Thread 139656747890432 has waited at dict0dict.c line 742 for 253.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
--Thread 139679657441024 has waited at buf0buf.c line 2527 for 253.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
--Thread 139656381863680 has waited at buf0buf.c line 2527 for 252.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
--Thread 139679663830784 has waited at dict0dict.c line 742 for 252.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
--Thread 139679664895744 has waited at dict0dict.c line 742 for 251.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
--Thread 139679655843584 has waited at dict0dict.c line 742 for 250.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
--Thread 139679662499584 has waited at dict0dict.c line 742 for 250.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
--Thread 139679661967104 has waited at dict0dict.c line 742 for 248.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1

--Thread 139679655311104 has waited at dict0dict.c line 742 for 248.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
--Thread 139679664363264 has waited at dict0dict.c line 742 for 248.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
--Thread 139679652116224 has waited at dict0dict.c line 742 for 248.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
--Thread 139679663564544 has waited at dict0dict.c line 742 for 247.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
--Thread 139679653447424 has waited at dict0dict.c line 742 for 247.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
--Thread 139679664629504 has waited at buf0buf.c line 2527 for 247.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
--Thread 139679657707264 has waited at buf0buf.c line 2527 for 247.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
--Thread 139679661168384 has waited at dict0dict.c line 742 for 247.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
--Thread 139679736440576 has waited at dict0dict.c line 742 for 247.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
--Thread 139679657973504 has waited at dict0dict.c line 742 for 247.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
--Thread 139679656376064 has waited at dict0dict.c line 742 for 247.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
--Thread 139679659570944 has waited at dict0dict.c line 742 for 247.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
--Thread 139679654512384 has waited at dict0dict.c line 742 for 246.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
--Thread 139679652648704 has waited at dict0dict.c line 742 for 246.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
--Thread 139679660369664 has waited at buf0buf.c line 2527 for 246.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
--Thread 139679651849984 has waited at dict0dict.c line 742 for 244.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
--Thread 139679658505984 has waited at dict0dict.c line 742 for 243.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
--Thread 139679735641856 has waited at buf0buf.c line 2527 for 243.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
--Thread 139679657174784 has waited at buf0buf.c line 2527 for 243.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
--Thread 139679664097024 has waited at dict0dict.c line 742 for 243.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
--Thread 139679658239744 has waited at dict0dict.c line 742 for 242.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1

InnoDB: ###### Diagnostic info printed to the standard error stream
InnoDB: Warning: a long semaphore wait:
--Thread 139679652914944 has waited at dict0dict.c line 742 for 272.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679655577344 has waited at buf0buf.c line 2527 for 271.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 139679656908544 has waited at buf0buf.c line 2527 for 271.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 139679656642304 has waited at buf0buf.c line 2527 for 271.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 139679654778624 has waited at buf0buf.c line 2527 for 271.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 139656811095808 has waited at buf0flu.c line 1481 for 270.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 139679652382464 has waited at buf0buf.c line 2527 for 270.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 139679661434624 has waited at buf0buf.c line 2527 for 270.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 139679663032064 has waited at buf0buf.c line 2527 for 270.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 139679665428224 has waited at dict0dict.c line 742 for 267.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139656747890432 has waited at dict0dict.c line 742 for 267.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1

InnoDB: Warning: a long semaphore wait:
--Thread 139679657441024 has waited at buf0buf.c line 2527 for 267.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 139656381863680 has waited at buf0buf.c line 2527 for 266.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 139679663830784 has waited at dict0dict.c line 742 for 266.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679664895744 has waited at dict0dict.c line 742 for 265.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679655843584 has waited at dict0dict.c line 742 for 264.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679662499584 has waited at dict0dict.c line 742 for 264.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679661967104 has waited at dict0dict.c line 742 for 262.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679655311104 has waited at dict0dict.c line 742 for 262.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679664363264 has waited at dict0dict.c line 742 for 262.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679652116224 has waited at dict0dict.c line 742 for 262.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679663564544 has waited at dict0dict.c line 742 for 261.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679653447424 has waited at dict0dict.c line 742 for 261.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679664629504 has waited at buf0buf.c line 2527 for 261.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 139679657707264 has waited at buf0buf.c line 2527 for 261.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 139679661168384 has waited at dict0dict.c line 742 for 261.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679736440576 has waited at dict0dict.c line 742 for 261.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679657973504 has waited at dict0dict.c line 742 for 261.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1

InnoDB: Warning: a long semaphore wait:
--Thread 139679656376064 has waited at dict0dict.c line 742 for 261.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679659570944 has waited at dict0dict.c line 742 for 261.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679654512384 has waited at dict0dict.c line 742 for 260.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679652648704 has waited at dict0dict.c line 742 for 260.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679660369664 has waited at buf0buf.c line 2527 for 260.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 139679651849984 has waited at dict0dict.c line 742 for 258.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679658505984 has waited at dict0dict.c line 742 for 257.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679735641856 has waited at buf0buf.c line 2527 for 257.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 139679657174784 has waited at buf0buf.c line 2527 for 257.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 139679664097024 has waited at dict0dict.c line 742 for 257.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679658239744 has waited at dict0dict.c line 742 for 256.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679736174336 has waited at dict0dict.c line 742 for 254.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679735375616 has waited at dict0dict.c line 742 for 253.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679654246144 has waited at dict0dict.c line 742 for 253.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679734843136 has waited at dict0dict.c line 742 for 251.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679653181184 has waited at dict0dict.c line 742 for 249.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679663298304 has waited at dict0dict.c line 742 for 249.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679661700864 has waited at dict0dict.c line 742 for 248.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1

InnoDB: Warning: a long semaphore wait:
--Thread 139656758646528 has waited at buf0buf.c line 2527 for 246.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 139679660103424 has waited at dict0dict.c line 742 for 244.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139656747624192 has waited at dict0dict.c line 742 for 242.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
InnoDB: Pending preads 0, pwrites 0
InnoDB: ###### Diagnostic info printed to the standard error stream
InnoDB: Warning: a long semaphore wait:
--Thread 139679652914944 has waited at dict0dict.c line 742 for 303.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679655577344 has waited at buf0buf.c line 2527 for 302.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 139679656908544 has waited at buf0buf.c line 2527 for 302.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 139679656642304 has waited at buf0buf.c line 2527 for 302.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 139679654778624 has waited at buf0buf.c line 2527 for 302.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 139656811095808 has waited at buf0flu.c line 1481 for 301.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 139679652382464 has waited at buf0buf.c line 2527 for 301.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 139679661434624 has waited at buf0buf.c line 2527 for 301.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 139679663032064 has waited at buf0buf.c line 2527 for 301.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 139679665428224 has waited at dict0dict.c line 742 for 298.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139656747890432 has waited at dict0dict.c line 742 for 298.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1

Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679657441024 has waited at buf0buf.c line 2527 for 298.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 139656381863680 has waited at buf0buf.c line 2527 for 297.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 139679663830784 has waited at dict0dict.c line 742 for 297.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679664895744 has waited at dict0dict.c line 742 for 296.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679655843584 has waited at dict0dict.c line 742 for 295.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679662499584 has waited at dict0dict.c line 742 for 295.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679661967104 has waited at dict0dict.c line 742 for 293.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679655311104 has waited at dict0dict.c line 742 for 293.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679664363264 has waited at dict0dict.c line 742 for 293.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679652116224 has waited at dict0dict.c line 742 for 293.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679663564544 has waited at dict0dict.c line 742 for 292.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679653447424 has waited at dict0dict.c line 742 for 292.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679664629504 has waited at buf0buf.c line 2527 for 292.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 139679657707264 has waited at buf0buf.c line 2527 for 292.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 139679661168384 has waited at dict0dict.c line 742 for 292.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679736440576 has waited at dict0dict.c line 742 for 292.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679657973504 has waited at dict0dict.c line 742 for 292.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679656376064 has waited at dict0dict.c line 742 for 292.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1

InnoDB: Warning: a long semaphore wait:
--Thread 139679659570944 has waited at dict0dict.c line 742 for 292.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679654512384 has waited at dict0dict.c line 742 for 291.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679652648704 has waited at dict0dict.c line 742 for 291.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679660369664 has waited at buf0buf.c line 2527 for 291.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 139679651849984 has waited at dict0dict.c line 742 for 289.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679658505984 has waited at dict0dict.c line 742 for 288.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679735641856 has waited at buf0buf.c line 2527 for 288.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 139679657174784 has waited at buf0buf.c line 2527 for 288.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 139679664097024 has waited at dict0dict.c line 742 for 288.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679658239744 has waited at dict0dict.c line 742 for 287.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679736174336 has waited at dict0dict.c line 742 for 285.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679735375616 has waited at dict0dict.c line 742 for 284.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679654246144 has waited at dict0dict.c line 742 for 284.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679734843136 has waited at dict0dict.c line 742 for 282.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679653181184 has waited at dict0dict.c line 742 for 280.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679663298304 has waited at dict0dict.c line 742 for 280.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139679661700864 has waited at dict0dict.c line 742 for 279.00 seconds the semaphore:
Mutex at 0x2486ba78 '&dict_sys->mutex', lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
--Thread 139656758646528 has waited at buf0buf.c line 2527 for 277.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 '&buf_pool->page_hash_latch'
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628

Version: '5.5.23-rel25.3' socket: '/tmp/mysql.sock' port: 0 Percona Server with XtraDB (GPL), Release rel25.3, Revision 240
InnoDB: Warning: a long semaphore wait:
--Thread 140479082170112 has waited at buf0buf.c line 2527 for 241.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 140479079507712 has waited at buf0buf.c line 2527 for 241.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 140502064256768 has waited at buf0buf.c line 2527 for 241.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 140479088293632 has waited at buf0buf.c line 2527 for 241.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 140479081903872 has waited at buf0buf.c line 2527 for 241.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 140479080572672 has waited at buf0buf.c line 2527 for 241.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 140479075780352 has waited at buf0buf.c line 2527 for 241.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 140479111403264 has waited at buf0flu.c line 1481 for 241.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
InnoDB: Pending preads 0, pwrites 0

=====================================
120603 21:00:07 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 30 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 519884 1_second, 519864 sleeps, 51784 10_second, 2305 background, 2305 flush
srv_master_thread log flush and writes: 551646
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 83453, signal count 157821
--Thread 140479082170112 has waited at buf0buf.c line 2527 for 247.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
--Thread 140479079507712 has waited at buf0buf.c line 2527 for 247.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
--Thread 140502064256768 has waited at buf0buf.c line 2527 for 247.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
--Thread 140479088293632 has waited at buf0buf.c line 2527 for 247.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
--Thread 140479081903872 has waited at buf0buf.c line 2527 for 247.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
--Thread 140479080572672 has waited at buf0buf.c line 2527 for 247.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
--Thread 140479075780352 has waited at buf0buf.c line 2527 for 247.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
--Thread 140479111403264 has waited at buf0flu.c line 1481 for 247.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
--Thread 140479100913408 has waited at buf0buf.c line 2527 for 246.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
--Thread 140479079241472 has waited at dict0dict.c line 742 for 246.00 seconds the semaphore:
Mutex at 0x24c5ea78 '&dict_sys->mutex', lock var 1
waiters flag 1

--Thread 140479077111552 has waited at buf0buf.c line 2527 for 246.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
--Thread 140479078708992 has waited at buf0buf.c line 2527 for 246.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
--Thread 140479088027392 has waited at buf0buf.c line 2527 for 245.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
--Thread 140502064789248 has waited at buf0buf.c line 2527 for 245.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
--Thread 140479086163712 has waited at buf0buf.c line 2527 for 245.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
--Thread 140479081371392 has waited at dict0dict.c line 742 for 244.00 seconds the semaphore:
Mutex at 0x24c5ea78 '&dict_sys->mutex', lock var 1
waiters flag 1
--Thread 140479089624832 has waited at dict0dict.c line 742 for 244.00 seconds the semaphore:
Mutex at 0x24c5ea78 '&dict_sys->mutex', lock var 1
waiters flag 1
--Thread 140502065055488 has waited at dict0dict.c line 742 for 243.00 seconds the semaphore:
Mutex at 0x24c5ea78 '&dict_sys->mutex', lock var 1
waiters flag 1
--Thread 140479090157312 has waited at dict0dict.c line 742 for 242.00 seconds the semaphore:
Mutex at 0x24c5ea78 '&dict_sys->mutex', lock var 1
waiters flag 1
--Thread 140479086429952 has waited at dict0dict.c line 742 for 242.00 seconds the semaphore:
Mutex at 0x24c5ea78 '&dict_sys->mutex', lock var 1
waiters flag 1
--Thread 140502004061952 has waited at dict0dict.c line 742 for 241.00 seconds the semaphore:
Mutex at 0x24c5ea78 '&dict_sys->mutex', lock var 1
waiters flag 1
--Thread 140479074449152 has waited at buf0buf.c line 2527 for 241.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
--Thread 140479084300032 has waited at dict0dict.c line 742 for 241.00 seconds the semaphore:
Mutex at 0x24c5ea78 '&dict_sys->mutex', lock var 1
waiters flag 1

InnoDB: ###### Diagnostic info printed to the standard error stream
InnoDB: Warning: a long semaphore wait:
--Thread 140479082170112 has waited at buf0buf.c line 2527 for 272.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 140479079507712 has waited at buf0buf.c line 2527 for 272.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 140502064256768 has waited at buf0buf.c line 2527 for 272.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 140479088293632 has waited at buf0buf.c line 2527 for 272.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 140479081903872 has waited at buf0buf.c line 2527 for 272.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 140479080572672 has waited at buf0buf.c line 2527 for 272.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
--Thread 140479075780352 has waited at buf0buf.c line 2527 for 272.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 '&buf_pool->page_hash_latch'
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628

Comment

How are your backups performed?
Could you have a SAN that takes snapshots of the disk?
Or a remote backupserver that locks files during backup?

I do not believe in coincidences and although there isn't anything in the local crontab. Things like backups (which are very often executed by a remote server) and sometimes some applications contain grooming or monitoring/software upgrade/patch solutions that are network wide run regularly.

Good luck!

Comment

Thanks for you suggestion - unfortunately this can't be a backup issue - I only take backups at 04.00 am using innobackupex.

That specific cron task performs only vBulletin (discussion forum software) thread views counting. "This script updates thread views once per hour. This option is suggested for forums with large numbers of threads or users.".

Of course this whole issue could be something else but it oddly coincides with that 56 minute mark (because that script runs every hour and these stalls have happened only at 8 pm 56min.

Hopefully somebody is able to go through those logs and explain what could cause this (I know that error is partly of because Innodb waits for something and then spits out that "--Thread 140479082170112 has waited at buf0buf.c line 2527 for 272.00 seconds the semaphore:" when it's 'tired' of waiting. BUT of course the big thing is that why it stalls the whole mysql until I kill -9 it.

I think my two options are 1) either to use that "innodb_lazy_drop_table" option or 2) instead of updating thread views on a hourly basis, update them immediately.

Can someone read the restarting logs and give an opinion if according to those logs during every restart tables were either ok, or repaired.
I'm just curious because the restart log is very different in first one compared to the rest "InnoDB: Progress in percents: 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed"
as the rest didn't have that. Is it because the transaction log was probably flushed during or just before that stall except in the first and second one?

"Modified age 30
Checkpoint age 30"
versus
"Modified age 0
Checkpoint age 0"