Using Gearman for Nightly Build and Test

At Tokutek, Rich Prohaska used Gearman to automate our nightly build and test process for TokuDB for MySQL. Rich is busy working on TokuDB, so I’m writing up an overview of the build and test architecture on his behalf.

Build and Test Process

Rich created a script, nightly.bash, that gets kicked off every night as a cron job. Nightly.bash creates a separate Gearman job for each build target. We have a separate build target (unique binary) for each combination of operating system (e.g. Linux, Windows, etc.) and HW architecture (e.g. i686, x86_64) supported by TokuDB. As we support more operating systems over time, the number of build targets grows quickly so we needed a build and test architecture that scales, and Gearman makes it easy.

Gearman then automatically distributes the build jobs to a set of systems set up as “Build Workers” with each available worker running a build for the specified build in parallel. For each build that completes, successfully, the resulting binary is stored in an Amazon S3 bucket and a regression test job is submitted to the Gearman job scheduler. Storing binaries in S3 costs less than checking them in to our hosted svn repository.

Test jobs are then distributed by Gearman to a set of “Test Workers” where the appropriate binary is read from Amazon S3 and regression tests are run. We currrently run mysql tests, sql bench tests, and TokuDB specific tests. New tests can easily be added by modifying scripts. Jobs submitted to Gearman specify a “function,” and workers specify supported functions when registering with Gearman. Functions are used to distribute build and test jobs to servers with the correct operating system and HW architecture.

Looking Ahead

Using Gearman, we built a highly flexible build and test infrastructure that easily scales to support more build targets and additional tests. Build and test workers can be added seamlessly to increase capacity and reduce the elapsed time required to finish the build and test cycle. Currently, everything runs on physical servers inside Tokutek but we designed the architecture to run in the cloud where virtual machines can be created on demand to support a large matrix of build targets and to complete large regression cycles quickly. All we need now is SSL support in Gearman to deploy on Amazon EC2…

You are correct; the blog had some errors. The build workers build MySQL binaries from source. The test workers run tests on the binaries. Gearman allows us to scale the number of workers as our test load increases. We are now running builds for 2 versions of MySQL (5.1 and 5.5) as well as Mariadb (5.1).