InnoDB errors after state transfer

07-17-2013, 05:01 AM

I have a problem with a four node cluster after a SST Transfer when a node has failed.

The facts:

- Four node Percona XtraDB Cluster running version 5.5.29-23.7.2.389.rhel6 on x86_64
- Using CentOS 6.4
- Both methods tried (rsync and xtrabackup)
- Sometimes large imports will break cluster as well. One random node stops and after rejoin the same error as below is received.
- Credentials for the transfer are correct in the whole cluster and the transfer works

When a node has joined the cluster again it requests a transfer, because it is out of sync. As far as I can see the transfer succeeds and the node has been joined the cluster again. When I try to select something from the transferred database I receive the following errors:

130125 15:57:16 InnoDB: Error: page 1 log sequence number 63541754
InnoDB: is in the future! Current system log sequence number 1611701.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.5/...-recovery.html
InnoDB: for more information.

or

130125 15:58:32 InnoDB: Error: table `xxxx`.`xxxx` does not exist in the InnoDB internal
InnoDB: data dictionary though MySQL is trying to drop it.
InnoDB: Have you copied the .frm file of the table to the
InnoDB: MySQL database directory from another database?
InnoDB: You can look for further help from
InnoDB: http://dev.mysql.com/doc/refman/5.5/...eshooting.html

or

130125 16:21:38 InnoDB: Error: trying to open a table, but could not
InnoDB: open the tablespace file './xxxx/xxxx.ibd'!
InnoDB: Have you moved InnoDB .ibd files around without using the
InnoDB: commands DISCARD TABLESPACE and IMPORT TABLESPACE?
InnoDB: It is also possible that this is a temporary table #sql...,
InnoDB: and MySQL removed the .ibd file for this.
InnoDB: Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/...-datadict.html
InnoDB: for how to resolve the issue.

which can result in:

InnoDB: Error: tablespace id is 4955 in the data dictionary
InnoDB: but in file ./xxxx/xxxx.ibd it is 4991!
130125 16:22:18 InnoDB: Assertion failure in thread 140335641454336 in file fil0fil.c line 776
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/...-recovery.html
InnoDB: about forcing recovery.
15:22:18 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=8388608
read_buffer_size=2097152
max_used_connections=12
max_threads=151
thread_count=10
connection_count=10
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 628597 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.