Test Suite for replication (rpl) made GTID agnostic

Global Transaction Identifiers (GTID)
was introduced in 5.6 and MTR tests were added specifically for testing
GTID. To improve test coverage we wanted to run all tests in rpl suite
with GTID ON. This was not possible because many will have result
differences and fail. We have addressed this in 5.7.4 and have added
regular regression runs of rpl suite with GTID turned on. Here is an
example command line that is run regularly:perl mysql-test-run.pl –force –timer –debug-server –parallel=auto
–experimental=collections/default.experimental –comment=rpl_gtid-debug
–vardir=var-rpl_gtid-debug –suite=rpl –mysqld=–enforce-gtid-consistency
–mysqld=–log-slave-updates –mysqld=–gtid-mode=on
–skip-test-list=collections/disabled-gtid-on.list –big-test
–testcase-timeout=60 –suite-timeout=360
Refer WL#7205 for more details.

Moved InnoDB compression tests to a separate suite

Tests for InnoDB compression were part of the InnoDB suite and hence
it was not easy to run only these tests with different compression
options. Compression tests are now moved to a new suite called
innodb_zip. This change is done from 5.5 onwards and following command
lines were added to regression runs.

Minor enhancement to mysql-test-run.pl

A new option “–do-test-list was added to mysql-test-run.pl . This
will take a file name as an argument and run the tests listed in the
file.eg: perl mysql-test-run.pl do-test-list=mytests.list$cat mytests.listfederated.federated sys_vars.all_vars main analyze main archive main blackhole
Above command will run all tests listed in mytests.list. This is
useful in development when we are working on a feature that will have
impact on tests spread across multiple suites. The relevant tests can
now be grouped together and run with a simple command line.

Thursday, March 6, 2014

This is a continuation of my post on migration-of-mtr-suites-to-use-innodb . To set the context here is a quick recap.
MySQL 5.5 had the following changes with respect to the default engine

Default engine in the server changed from MyISAM to InnoDB

MTR modified to start server with the old default MyISAM. (This was
required because historically most test results were recorded with
MyISAM and many of them would fail if tests were run with the new server
default, InnoDB)

Tests retained as is in 5.5 and planned to migrate them to run with default engine in a future release

In MySQL 5.7 release we have started the migration project. Right in
the beginning we realized that the switch of default engine had an
unexpected side effect. MTR tests that were developed post 5.5 were also
getting recorded and run with MyISAM unless an engine was explicitly
specified in the test. I.e. we were in a state where not only old
tests, but also new tests were running with the old default engine
MyISAM. As a result the backlog of tests that had to be migrated was
growing over time. We decided to fix the problem with new tests first
and old tests later.

The only way we could force new tests to run with InnoDB by default
was to remove the engine switch in MTR. This however would bring back
the problem of existing tests failing with result difference. We decided
to tackle this problem with a different approach. Instead of switching
the engine in the test tool itself, it was switched in the existing
tests. This is where the force_myisam_default.inc makes its entry. If
this file is included it will tell MTR to start server with MyISAM. It
was added to all existing tests that did not have any explicit engine
specification at the test level. To be more precise it was added to
tests that did not have have_innodb.inc, have_archive.inc etc. We then
removed the switch in MTR, so that by default it starts the server with
its default engine which is InnoDB. With these 2 changes we got to the
state where new tests by default were running with InnoDB.This was a big
step forward and on hindsight I think we should have done this instead
of the engine switch in 5.5 .

Having fixed the problem with new tests, we have now shifted our
focus to migration of old tests. The migration project does not in any
way reduce the test coverage on MyISAM; rather it just adds InnoDB
coverage where it was missing. Post migration tests for features which
are dependent on the engine will have MyISAM and InnoDB variants. Tests
for features that are expected to have same behavior on all engines may
run only with the default engine InnoDB.
We have outlined following strategy for the migration project:

Maintain MyISAM and innodb variants for tests and subtests that are dependent on the engine

Remove MyISAM dependencies from tests that are not meant for testing MyISAM features

Retain tests for MyISAM only features in MyISAM

There is an element of risk in this project since any small mistake
can lead to missing or even worse wrong tests. However be rest assured
that this is being handled with utmost care by our strong QA team. To
ensure correctness and stability all result differences are analyzed and
discussed with the concerned people before making any changes. Some of
the differences were due to bugs and these are fixed before test
migration. Some others are expected and are accepted as they are. Listed
below are a few examples of the changes made to the result files:

As of 5.7.3 release migration is complete for auth_sec, federated,
funcs_1, funcs_2, jp, opt_trace, perfschema, stress and sys_vars suites.
It is not all done yet and you will see more and more tests getting
migrated in the upcoming milestone releases.

If you have any questions or feedback regarding this project please
post them here. I would love to hear what the community thinks about
this.

Thursday, January 30, 2014

In MySQL 5.7.2 a new include file “–source
include/force_myisam_default.inc“ made its appearance in most of the
.test files in the MTR suite. If you were wondering about this read on.
I will explain this change in two blogs. The first will describe why
we are doing this and the next will explain how it is being done.

In order to set the context let me delve a bit into history. Starting
from MySQL 5.5, the default storage engine for new tables is InnoDB.
This change was coupled with a reverse switch in mysql-test-run, which
made MyISAM the default storage engine for the server started through
MTR(mysql-test-run). As a result default storage engine in the server
was innodb, but most tests were run with the old default MyISAM.

Let me explain why such a switch was required. The usual practice in
MTR test development was to not specify an engine in create table
statement, unless the test is specifically for a non-default engine like
merge, archive, innodb etc. As a result tables in most tests were
created with the default engine which is MyISAM. From a test coverage
perspective this was a perfectly sound strategy. Tests for engine
specific properties were run with the appropriate engines and tests for
engine independent properties were run with the default engine, MyISAM.
However this strategy became a problem when the default engine was
changed. Tables in tests that do not specify any engine started getting
created with the new default innodb leading to result differences and in
turn test failures. There were hundreds of failing tests and fixing all
this was an enormous task.

To overcome this situation following 2 options were considered:

Option 1 : Migrate all tests to run with innodb engine. This would
involve analyzing all test failures and modifying the test or result
files as required.

Option 2: Switch the default storage engine in MTR so that there is no need to change the test or result files.

Option 1 would have been the ideal, but it became quickly obvious
that this will be a time and resource intensive solution. On the other
hand option 2 was easy, but risky. A careful evaluation of the test
suite showed that the risk with option 2 is not very high. The tests
that did not have any engine specified were either tests for MyISAM only
features or tests that were not considered to be sensitive to the
engine. For everything else there were engine specific variants of the
test. For example the partitioning test has 6 flavors –
partition_archive, partition_federated, partition_csv, partition_innodb,
partition_blackhole and partition_myisam . So the risk with option 2
was not very significant and was acceptable at least in the short term.

After evaluating the pros and cons of both options it was decided to
go with option 2 for 5.5 release and to implement option 1 in a future
release. MySQL 5.5 and 5.6 were released using MyISAM as the default in
MTR suite. In 5.7 we have started implementing option 2 and this is is
the reason for the force_myisam_default.inc.

Did I confuse you? I said migrate to innodb and the inc file says
force_myisam_default. Well, if you are confused wait for my next blog