LRU_list_mutex contention - Percona server 5.1.54

03-03-2011, 07:24 PM

We have run into problems with mysql community edition 5.1.53, our DB traffic seems to be serialized quite often into a single thread concurrency. For example, 5 more demanding SELECT queries are running in the processlist, but the HW is idle (1 CPU core used, loadavg 1.2, nearly no disk usage). This seems to be worse when exectuting some more demanding SELECT queries, OLTP workload is fine.

I examined SHOW INNODB STATUS and have seen lot of buffer pool mutex contention in the SEMAPHORES section (buf0buf.c).

I have read XtraDB has implemented some split of this mutex, but after upgrade to Percona server (5.1.54-rel12.5-log) , the situation is not better and I see this:

Is it something standard, or should I blame it on some HW problems? Is there any testcase like sysbench, which could find any problems?
The system is supermicro 2x Intel X5670 (24 cores including HT), Intel SSD drives + HDD RAID for logs. 70GB buffer pool, disabled adaptive hash index, disabled query cache, 4000MB innodb logs in total.

I have tried setting innodb_thread concurrency to 12, 20 and zero, but without any effect. Setting CPU affinity to the first 12 cores makes no difference. OS is CentOS, kernel 2.6.18-194.32.1.el5.

Comment

Comment

Gmouse, innodb_buffer_pool_instances is available since 5.5, am I right? I still don't have a feeling that 5.5 is stable enough. Percona claims that they solved the buffer pool mutex contention in another (better) way. I wonder if they meant XtraDB in the 5.1 or 5.5 release.

I think compression is useless until innodb:
- makes it scale well
- makes alter table multithreaded. If you need to compress some tables, you do it because they are large. If it takes 20+ hours, the maintenance becomes a nightmare.

Comment

krteq, that is a very interesting finding. I would not be surprised if we can solve this bug for you. Please contact our sales department at http://www.percona.com/contact/sales/ if this is something you are interested in having us fix.