Site Navigation

Disclaimer

The views or opinions expressed on this blog are my own and do not necessarily reflect the views or opinions of my current employer. The views or opinions expressed by visitors on this blog are theirs solely and may not reflect mine.

Probably the biggest highlight of 0.15 is the addition of zbackup as an additional backup type. I'd like to thank Ivan Korjavin for contributing this new feature.

Additionally, this release provides several improvements and bug fixes. For example, it's now possible to back up more than a single my.cnf configuration file by providing a directory name like /etc/mysql instead.

Some notable highlights from the ChangeLog (please check the bzr commit log for a more detailed history of changes):

Improved error handling: send out an email if mail_report_on has been set to "errors" and a log message with log level LOG_ERR was logged.

Abort and clean up if the snapshot volume could not be mounted

Abort and clean up if the backup creation failed

Exit with a non-zero return code if errorstate was set

Enabled timestr() formatting sequences for snapshot volume names, to create snapshot names which contain a dynamic date value. This can be useful when thin provisioned snapshots are used as the actual backup by enabling the keep_snapshot option

Added new configuration option "recoveryopts" that allows to modify the command line options that are passed to the MySQL instance that performs the InnoDB recovery (BUG#544295 and BUG#1091950). The current default options are "--skip-networking --skip-grant --bootstrap --skip-syslog --skip-slave-start" (which better accommodates MySQL 5.1)

Merged patches from Alexandre Anriot to provide a mail reporting method which sends the log output via email and a new "purge" action that allows to only keep a defined number of backups in the backup directory

Applied patch contributed by Klaus Ethgen that removes the annoying LVM warnings about a leaked file descriptor (DBD:mysql does not add FD_CLOEXEC on the MySQL socket)

BUG#1086313: mvlvmbackup lacks support for 'thin' LVM snapshots (Thanks to Neil Wilson for the patch)

BUG#787063: Ensure to clean up temporary .pos files when the snapshot mount is kept, too

Applied small patch from Norbert Tretkowski: change umask to 077 before creating the tarball backup files to prevent them from being readable by everyone

BUG#593220: Fixed the compressarg option for cat (removed the comment)

Applied patch contributed by Ben Bonnel to add a new configuration option "rsnaprsyncarg" that allows one to pass special options to the rsync process when using the rsnap backend.

Applied patch provided by Ask Bjørn Hansen to allow targets not to have a timestamp (e.g. for rsync targets)

Thursday, November 5. 2009

This blog post is a by-product of my preparation work for an upcoming talk titled "Why you should be using a distributed version control system (DVCS) for your project" at SAPO Codebits in Lisbon (December 3-5, 2009). Publishing these thoughts prior to the conference serves two purposes: getting some peer review on my findings and acting as a teaser for the actual talk. So please let me know — did I cover the relevant aspects or did I miss anything? What's your take on DVCS vs. the centralized approach? Why do you prefer one over the other? I'm looking forward to your comments!

Even though there are several distributed alternatives available for some years now (with Bazaar, git and Mercurial being the most prominent representatives here), many large and popular Open Source projects still use centralized systems like Subversion or even CVS to maintain their source code. While Subversion has eased some of the pains of CVS (e.g. better remote access, renaming/moving of files and directories, easy branching), the centralized approach by itself poses some disadvantages compared to distributed systems. So what are these? Let me give you a few examples of the limitations that a centralized system like Subversion has and how these affect the possible workflows and development practices.

Sunday, September 6. 2009

I am happy to announce that mylvmbackup version 0.13 has now been released. This release includes a fix for a nasty bug in on of the recently added Perl hooks (precleanup.pm) and some added functionality (better support for remote rsync backups).

From the ChangeLog:

Deleted sample precleanup.pm hook as it has potential to cause harm and is too specialized on a particular use case (BUG#394668)

Saturday, June 20. 2009

After a long hiatus, I am happy to announce that mylvmbackup version 0.12 has now been released. This release includes a large number of improvements, minor code cleanups, as well as some new functionality. In particular, I would like to thank Matthew Boehm, Tim Stoop, Baron Schwartz, Ville Skyttä and Ronald Bradford for their contributions.

Thursday, February 26. 2009

Today I gave my first MySQL University session as a speaker, talking about Backing up MySQL using file system snapshots. The talk went quite well (at least that was my impression) and we had ~10 people attending. The slides (PDF) and a recording of the session are now available from the Wiki page. Unfortunately the recording lacks the audio track, which is a bit of a bummer. We've submitted a support request with the DimDim folks, so hopefully they can provide us with a complete recording.

There was one question during the session that I was not able to answer myself, so I'm asking for your insights here:

Consider we're using InnoDB and MyISAM tables on a file system that can be snapshotted (e.g. Linux LVM or ZFS) and we're performing the following operations:

FLUSH TABLES WITH READ LOCK (yes, this won't help for the InnoDB tables)

Create the snapshot

Store the output of SHOW MASTER/SLAVE status in a file to be part of the backup

UNLOCK TABLES

Mount the snapshot

Start a second MySQL instance that accesses the tables on the snapshot, let InnoDB perform its table recovery

Shut down the second instance and perform the backup of the snapshot

The question that came up was if this actually still is a consistent backup, considering that InnoDB rolled back uncommited transactions. Does the state of the tables still match the binary log positions we noted before? I assume yes, as long as the transaction does not involve modifications non-transactional tables.

Another suggestion that came up was to change InnoDB's configuration variable innodb_max_dirty_pages_pct to "0" prior to performing the snapshot, to minimize the amount of dirty pages that have not been written to disk (and thus reducing the time required for recovering later). I wonder if this would make a difference...

What other InnoDB variables might have a noteworthy effect in the context of snapshot backups? I am looking forward to your comments.

Sunday, February 22. 2009

Sorry for the downtime of this site - until around a week ago I hosted my home page on a trusty Genesi Pegasos II system (powered by a PowerPC G4 Processor clocked at 1GHz, using Debian 4.0 PPC with 512 MB of RAM), serving these pages from my home DSL connection. Unfortunately this system provided no means of redundancy - the hard disk drive died.

Luckily I perform frequent backups, so I moved most parts of the site to a shared hosting space now - the picture gallery is unfortunately too big to fit into the space that I have there. I'll try to move the pictures into my Flickr account instead, but this will take some time.

Note that the primary domain name of this site is now lenzg.net - lenzg.org, (the domain that I tried to promote as the official domain for my site) used to redirect to the home machine at lenz.homelinux.org. Both now redirect to the new address instead. I've initiated the move of the lenzg.org domain to the other provider as well, so soon this site will be available from both the .org and .net domain. Please don't link to lenz.homelinux.org anymore, as that site will eventually go out of service. Until then, a small openSUSE Linux box (Intel PIII, 500 MHz, 192 MB of RAM) running lighttpd will perform the URL redirection.

Monday, December 1. 2008

Some days ago, I released version 0.11 of mylvmbackup a Perl script that performs consistent backups of a MySQL server by using LVM filesystem snapshots. The source archive as well as a generic RPM can be found on the project home page, packages for many Linux distributions are available on the openSUSE Build service.

This release includes some new functionality as well as numerous bug fixes and improvements, most notably:

Fixed Bug #271671: "overloading parameters does not work" by removing the default values for host and port from the configuration file and removing the unnecessary check for passing both host and socket at the same time. Updated documentation and configuration file comments accordingly.

Code cleanup: build up long command strings in a $command variable before passing it to system()

Renamed subroutine create_snapshot() to create_lvm_snapshot()

Merged patch from Matthew Boehm: Removed old asciidoc documentation in favor of POD style. This removes the dependency on the external program a2x for creating documentation and uses the 'built-in' pod2html and pod2man instead. Updated the Makefile to accommodate the change.

Applied patch from Matthew Boehm to make the backup file name suffix configurable via a "--suffix" option. Updated the man page accordingly.

Applied patch from Matt Lohier to support rsnap as a backup backend

Moved the list of contributors from the man page into a separate CREDITS file, added missing names

Saturday, September 20. 2008

I am happy to announce that mylvmbackup version 0.10 has been released.

You can download the updated package from the project home page or via the openSUSE Build Service.

This version fixes some bugs and includes new functionality:

Applied patch from Marc Haber: added option --keep_snapshot that will skip the removal of the backup snapshot before terminating the script. Providing the option --backuptype=none will now skip creating a backup using the builtin backup modules. Both options provide more flexibility when using hooks for performing the actual backup tasks or when the snapshot is considered to be the actual backup.

Added two new hooks: "backupsuccess" and "backupfailure" which are called respectively upon success of failure of the backup operation (Bug #264089)

Make sure that binaries are being found ($PATH may not include /sbin when called from cron), added missing entry for "lvs" to mylvmbackup.conf (Bug #255703)

Friday, July 11. 2008

I am happy to announce that a new version (0.9) of mylvmbackup has been released. This is the first release since the source code has been moved from Subversion to Bazaar and is now hosted on Launchpad.net. I would like to thank Robin H. Johnson and Patrick Hahn for providing the patches that contributed to this new release!

mylvmbackup is a tool for quickly creating backups of MySQL server's data files. To perform a backup, mylvmbackup obtains a read lock on all tables and flushes all server caches to disk, makes an LVM snapshot of the volume containing the MySQL data directory, and unlocks the tables again. The snapshot process takes only a small amount of time. When it is done, the server can continue normal operations, while the actual file backup proceeds.

This will hopefully make it easier for contributors to work on the code and share their modifications with others, removing me as the bottleneck for applying and testing patches for new releases. I chose Bazaar primarily because I wanted to get some more hands-on practice with it, now that the MySQL Server source trees have been transferred to it as well (see Kaj's announcement for details).

As mylvmbackup is closely related to the MySQL Server project, it made sense to choose the same platform and enjoy the cross-pollination effects and the infrastructure that Launchpad provides. Additionally, the distributed nature of Bazaar makes it more convenient to work with the code history and commiting changes locally without having to be online and connected to the SVN server.

I am sure that other DSCMSs like Git, Mercurial or darcs would have done the job equally well - nowadays it's very hard to choose

The "trunk" branch is now hosted on Launchpad. I assume that I will soon open up a development branch, that will receive heavier modifications first. I also plan to use the site for bug tracking and keeping track of feature requests (via Blueprints).

To create a local branch of the "trunk" repository, you can use the following command:

LVM already lets administrators create snapshots, but its design has the surprising property that every block you change on the original volume consumes one block for each snapshot. The resulting speed and space penalty usually makes the use of more than one or two snapshots at a time impractical.

Zumastor keeps all snapshots for a particular volume in a common snapshot store, and shares blocks the way one would expect. Thus making a change to one block of a file in the original volume only uses one block in the snapshot store no matter how many snapshots you have.

Replication

Andrew Tridgell's rsync is a wonderful tool for replicating files remotely. However, when doing periodic replication of large numbers of infrequently changing files, the overhead for figuring out what files need to be sent can be extreme.

Zumastor keeps track of which block change between one snapshot and the next, and can easily send just the changed blocks. Thus Zumastor can do frequent replication of large filesystems much more efficiently than rsync can.

I assume it's not ready for production use yet, but it would sure be interesting to investigate on how to utilize it for the purpose of running MySQL on top of it...

I will keep an eye on this project, I wonder if I will have to add support for Zumastor snapshots to mylvmbackup at some point?

Friday, April 11. 2008

I am happy to announce the release of mylvmbackup version 0.8. mylvmbackup is a tool for quickly creating backups of a MySQL server's data files. To perform a backup, mylvmbackup obtains a read lock on all tables and flushes all server caches to disk, makes an LVM snapshot of the volume containing the MySQL data directory, and unlocks the tables again. The snapshot process takes only a small amount of time. When it is done, the server can continue normal operations, while the actual file backup proceeds.

Below is the list of changes since version 0.6. You may wonder what happened to version 0.7 - it had a rather short life cycle as I was informed about a bug that I fixed quickly before I made a wider release announcement of 0.7.

Fixed a bug in the InnoDB recovery function: the second mysqld process clobbered the socket file of the primary MySQL instance (thanks to Alain Hoang for reporting this)

Updated the man page, noted some other limitations of the InnoDB recovery function

Added option "--skip_mycnf" to skip including a copy of the MySQL configuration file in the backup, added a safety check that the file actually exists prior to backing it up.

Updated package are available from the home page and via the openSUSE Build Service as usual. Updated packages for Debian/Ubuntu and Gentoo Linux should also be available shortly. Enjoy!

Speaking of LVM snapshot backups: I will be giving a talk about this subject at our MySQL Conference 2008 in Santa Clara, CA next week. If you are curious about how MySQL can be backed up using this technology, please consider to stop by!