Why?

If you run Drupal sites, you need some way to manage them; some way to keep them up-to-date. As of Drupal 7, there’s a built-in update manager, but it doesn’t use a VCS.

Most likely, you want use a VCS to manage your Drupal site. You may be downloading tarballs and checking their contents into your VCS, or maybe you’re using git submodules or even git-subtree. For all of these, there’s a whole lot of process, a lot of steps, a lot to learn, and a lot of tricky things you have to think about each time you want to update something. Too much that can go wrong.

That is fine for big, commercial, team-backed sites. But it’s a pain to do for smaller sites, that maybe only are updated once every few months. When you update rarely, it becomes very easy to forget about one of those tricky things. And then, for fear of doing X or Y wrong, which might break your site, it causes you to update your site even less often.

I probably evaluated everything out there1. All of it was too complex for my needs. I want my code deployment system to be simple and preferably Drupal-agnostic.

It’s that niche that this approach to Drupal site management attempts to simplify.

How?

mr is a small, stable shell script (written in Perl, unfortunately) to help manage multiple (VCS) repositories simultaneously. By using the Drupal plug-in provided by this project, you can leverage a subset of mr’s functionality to simplify Drupal site management.

The Drupal plugin offers you a simple, declarative syntax to automatically clone and update git repositories for each Drupal module you use from git.drupal.org.2

There’s almost nothing to learn. Hence, there’s almost nothing to forget. There is a single configuration file that is (mostly) declarative, so if you look at it again in 2, 6 or 12 months, it will still make sense.

Essentially, you only need to know two things:

the desired state is declared in the .mrconfig file

get to the desired state by running mr update

Note that the tool/approach described does not cover running database updates etc. (though it is possible to integrate that as well). I prefer to do this manually on smallish sites.

Getting started

Steps to be repeated on each machine:

Install mr. (Package in Debian, unofficial RPM and on OS X via brew.)

Install the Drupal plug-in for mr: copy the drupal file in the mr-drupal repository into /usr/share/mr/.

Creating a config file for your site

Steps to be repeated for each Drupal site:

Create a .mrconfig file (don’t forget the leading period!), it should look like this:

# Use the Drupal plug-in for `mr`.
[DEFAULT]
include = cat /usr/share/mr/drupal
# Drupal itself.
[www]
project = drupal
version = 7.19
# Set the proper file system permissions on all Drupal files. The owner is
# set to the current user, the group is set to www-data.
# See http://drupal.org/node/244924.
fixups = drupal_set_permissions `whoami` 'www-data'
# A branch of a Drupal module.
[www/sites/all/modules/cdn]
project = cdn
version = 7.x-2.x
# A tag of a Drupal module.
[www/sites/all/modules/comment_notify]
project = comment_notify
version = 7.x-1.1
# Custom themes live in a non-git.drupal.org repository. `mr` is smart enough
# to automatically know this is a git repository, and knows how to update it.
[www/sites/wimleers.com/themes]
checkout = git clone git://github.com/wimleers/wimleers.com-themes.git themes

As you can see, it’s easy to mix in custom modules/repositories.

Run mr update. Your Drupal site will be built based on your .mrconfig file, relative to where the .mrconfig file lives.3

Optionally check in the .mrconfig file into a VCS, so you can roll back to previous versions. Highly preferable, but not essential.

In the above example, we leveraged one “advanced” feature of mr: the ability to do “fixups” (after each checkout/update). In this case, to guarantee correct permissions using the drupal_set_permissions() function. mr-drupal also includes a git_apply_patch() function to git apply patches (it also automatically detects whether the patch has already been applied).

Note that there is nothing Drupal-specific about any of this! You can check out Drupal modules at www/sites/all/modules/, a static “status” site at www/status, HTML5 presentations at www/talks/, and so on. Things won’t break because Drupal or Drush are updated.

To update your site

Modify the .mrconfig file: change version numbers, add modules. Remove a module by adding a line like this: deleted = true.

In lib, you can define functions you want to use. In this case, we defined the get_dtap() and drupal_fs_group() functions. The latter uses the former.
Now the group used for enforcing the correct permissions is set based on the environment.

But really, you can do whatever you want: anything that can be used in a shell script can also be used here. This is the part of mr-drupal where you can make things as complex as you’d like. The complex parts are opt-in.

Conclusion

One file (.mrconfig) and one command (mr update) together cover 90% of the smallish Drupal site management needs.

P.S.: I’ve been using mr-drupal to keep my sites updated for the past several months without problems. Rather than spending >1 hour (to update locally, then look up drush commands and triple-check everything before calling drush rsync), it now takes me just a few minutes to update a site!

As is the case for any site that has its modules on disk as git repositories: you must use the Git deploy module to ensure Drupal core’s “Available Updates” report continues to work. ↩︎

Speed it up by telling it to work in parallel: mr -j 10 update would do up to 10 updates in parallel. ↩︎

The only exception: those that are marked as deleted. mr does not actually delete them for you. For example, if you’ve marked the devel and drupad modules as deleted and the devel module no longer exists, it’ll output something like this:

Why did you decide that git subtree was more complex that submodules? To me it looks less complex.

I presume you discounted Aegir because you have to have dedicated server/s?

One thing I really like about your approach is the declarative nature. I always want meta information that is separate from the information under control. It makes it easy to know if there have been changes which have not been processed; and, it lets you keep a history of changes. That’s one of the I like about using git. What I don’t like about git all the other systems I’ve found is the lack of space for information about the ‘special stuff’. The infrequent things you do for a particular site which need some sort of special attention - e.g. not updating a particular module.

Since 1.7.11 git-subtree has been part of mainline git, I think. That’s May 2012. Yes, it’s an extra set of commands in that you have to notify git when you are adding a Drupal module - you add a remote repo for the module pointing at drupal.org, then you do something like this to add a module:
git subtree add -P docroot/sites/all/modules/devel upstream-drupal-devel --squash -m "Adding in Devel 7.x-1.3" tags/7.x-1.3
After that if you make patches to devel, you have to remember when to use -P docroot/sites/all/modules/devel to notify git that the change applies to the devel module. So you have to have a commit for changes to the module. But you probably would anyway because a change to a module is a discreet element.

Where it falls down:

No drush integration. (But git, drush and mr updates are all different ways of tackling the same task. And a very quick look at drush yesterday made me think it wasn’t a ridiculous idea to add a flag to tell drush that’s how it should pull from drupal.org. )

git_deploy doesn’t work. No .info module version numbers (I’m wondering about a git hook. See https://support.mayfirst.org/wiki/faq/gitification which has an idea for .gitinfo. I’m imagining taking the version info from .mrconfig or the git subtree command and writing it into a .gitinfo.)

The declarative aspect of a .mrconfig or drush makefile gives a fixed point to navigate from in terms of updates, and there’s none of that. (But it looks like mr commands are not that hard and you’ve done the heavy lifting, so maybe mr+git-subtree.)

Special stuff: It’s the mixture of stuff you forget is special about a site when you don’t deal with it regularly. E.g. Remembering there’s a patch to a module that’s not worth getting into the drupal.org version of the module because it’s site-specific; remembering Hacked is installed to remind you about a change like that; holding back on upgrading a particular module because later versions lack some features - e.g. the D6 nodewords module that lost token support; site-specific rules to remember - e.g. a site I worked on a few years ago had its own template engine for historical reasons - before a module could be updated it needed a diff to check whether the module template files had changed - and if they had, then it needed work in preparation for the update.

When you use drush you don’t have a place for info like this. Neither does normal git. But mr does - the .mrconfig which you have to edit to update.

Caveat: I’m still experimenting with git subtree workflow, and I’m pretty sure I haven’t discovered all the gotchas.

Right, even when git-subtree is integrated in mainline git, it is still too cumbersome to use.

Oh, great point! I hadn’t even considered that yet. I guess it’d fall upon the git-subtree user to manually modify each git-subtree‘d-in Drupal module to add the module’s version number to the module’s .info file. That’s of course a rather insane requirement.

RE: “specialty stuff”: exactly! For this very site (<wimleers.com>), I have the Google Webmaster HTML file and one or two Drupal core patches in the git repository where I keep the .mrconfig file for this site. I then just let mr respectively copy it into the appropriate location and apply the patches :)

Thanks for your feedback — your point about “no good place for the metadata/special stuff” is a good one that I failed to mention!

I can answer that one from experience. If you have customizations of even one module and one theme, a regular (let’s say monthly) automatic Drush update can mean monthly havoc for your site. If done every 6 months, it means in 6 months you have to be poised to undo and repair whatever the auto -update caused. Having it pre-scripted can be very handy so it will just avoid the custom modules. And if nothing else, the script itself serves as a reminder to revisit the customizations to see if they’re still necessary.

It was high on my to-do list with my admin to find an alternative to automatic Drush updates, so, Wim, thanks!! I’ll be bringing this to his attention!

It is not explicitly supported to be used for updating. Drush maintainers argued almost for the entire lifetime of the relevant issue that it should not be supported (comment #78, May 2, 2012). Up to that comment in that issue, this was the maintainers’ POV: “For the record, it is still my opinion that this issue should be closed ‘will not fix’.” Two comments later, on May 22, 2012, it turns out Drush 5 already “kinda supports this”. One dangerous bug/quirk was noted one comment further, and yet another comment further that quirk got fixed in another issue. Then the issue was closed and marked as done. As you can see, it has been marked to “kinda work”, but has no explicit maintainer support/approval. If a maintainer thinks it’s a bad idea and does not explicitly support it … then who will support it when you run into problems?

Drush frequently breaks things when a new Drush version is released (or maybe I’ve just been using the parts where that is the case).

It is highly Drupal-specific. I like to be able to easily pull in non-Drupal stuff too, and have that be updated as well.

It is a very big, complex beast: Drush 8.x-6.0-rc1 is ±2 MiB (±500 KiB gzipped) MiB. I’ve worked on getting a new feature in, and it was a most painful experience. It is better to rely on simpler tools, which one is able to modify to suit needs if necessary.

Combine all of the above, and it simply boils down to this: I don’t trust Drush for deployment. Sure, I could start to experiment with it again, possibly Drush 6 is better and hence sufficiently reliable for doing deployment. But Drush only becomes more complex, so it becomes increasingly hard to evaluate it.

Drush can be used for updating. See
https://drupal.org/sandbox/grndlvl/1930260
We’ve been using this in-house for all of our major projects and it’s great. We have a build server running on every commit to Git that uses make-sync to confirm that what is committed in Git matches what is defined in the make file.

I haven’t really experienced much breakage with Drush moving up in versions. But I think that’s because we’ve been fairly conservative - we have a ‘standardized version’ that is on all of our machines and we don’t upgrade it very often.

I am 100% with you on the complexity issue though. I’ve done a bit of work on Drush plugins and I always get the feeling that something is really wrong, mostly with the argument system. But it’s big because it does a lot more than just library/project management. Mr. can’t transfer DBs from server to server with some sanitization thrown in (well it could, but it would require an equal amount of code).

You mention the word “deployment”, but that’s not really what we’re talking about here. That’s a completely different problem, but I agree — I don’t see Drush as being useful there.

https://drupal.org/sandbox/grndlvl/1930260 states that it’s an experimental project, and it requires a patch to Drush, andmake_sync.drush.inc is hardly trivial at >650 LoC that are extremely hard to understand without deep Drush knowledge.

It’s hard to figure out how to use it effectively, because it is not explicitly and officially supported (see my above comment). That’s precisely why somebody had to go and build Drush Make Sync in the first place. That’s why you are using Drush Make Sync, rather than plain Drush.

I had never heard of Drush Make Sync, otherwise I’d given it a try for sure!

I see a lot of value in staying within the Drupal/Drush ecosystem. I’m not opposed to Drush. I’m only saying that Drush/drush make don’t work for smaller use cases — I said this in the article:

That is fine for big, commercial, team-backed sites.

And in fact, many of the things are in exact line with my argument:

You’re using Drush Make Sync, not Plain Drush Make.

Drush Make Sync was developed/sponsored by a company (your company).

You’ve been using this “in-house”, which kind of implies there hasn’t been any attempt to make this the standard way across all of Drupal: it’s still a sandbox project, there haven’t been any blog posts, and seemingly no attempts to make this part of Drush.

You have a build server.

You’re conservative in upgrading Drush, which implies there’s a process or at least a lot of testing before updating.

All of those points require a big team, because a one-person project is never going to figure out/build all of the above!

Actually, I mean anything. Like e.g. cron scripts, HTML5 presentations, a subsite. Anything. Much more than 3rd party libraries that are used by Drupal modules.

Finally: yes, mr and mr-drupal don’t solve the whole deployment problem, they only aid in solving the code deployment problem.
In my case, it’s good enough to deal with DB dumps and moving those around manually. Small sites have no need for two-way content syncing and all that.

It’s not just that Drush provides both code and DB deployment, it’s that it does even more than that. That’s why it’s so complex. That’s why it’s so big. That’s why it’s scary to work with.

If something pops up that’s equally simple as mr-drupal, or only slightly more complex, and it deals with DB deployment, then I’d be happy to switch to that. I don’t see that happening soon though.
Even when that does happen, I’ll still be able to use mr for handling other things — it is in no way specific to Drupal or even web development. Many people use it to manage their home directory, for example.

So you must be referring to https://github.com/wimleers/mr-drupal/pull/1 :) Thanks for starting that! But, indeed, I think that’s not really the right approach. For starters, you’ll run into problems when updating a module from one version to the next, because you’d have to clean up the module’s .info file.
I think it’s better to just fix the performance problem in the git_deploy module: https://drupal.org/node/1511112.

I can’t say that I have. I saw it pass by, but I completely forgot about it. Thanks for reminding me!

Kraftwagen again seems too complex for simple sites. Too many steps to remember. It also aims to do content & config migration/syncing, which is much, much more than mr-drupal does.

Also, config syncing will be a solved problem in Drupal 8, so a big part of what Kraftwagen does will become irrelevant then.

I’d say that mr-drupal should not be used for sites that need content migration. For those, Kraftwagen seems a much better solution. It is possible to use mr-drupal in these situations too, but then you’ll need to think of a content migration strategy yourself — or maybe just use that part of Kraftwagen (though I’m not sure if that’s feasible).

While there is no real silver bullet when it comes to deployment, I think this is a very handy tool for small to medium projects.

The big advantage here is that there’s less code to worry about as the state of a site instance gets dictated by a declarative config file, not by the code checked into the project repo. It does remind me of tools like Chef or Puppet, which pretty much implement the same concepts. :-)

Quick questions:

What does your project repository contain when you rely on a tool like mr? I guess you could reduce this to just the bare customs:

.gitignore

.mrconfig

.htaccess (if modified)

sites/*/{modules|themes}/{custom|contrib_patched}

Am I missing anything else?

I think the .mrconfig file can be used to automate quite a few other things as well:

Permissons on the files/ folder

Apply patches and move patched modules out of contrib/

I’ve yet to implement it in a workflow, but this looks quite promising.

no.gitignore file, because it doesn’t seem very useful to me at the .mrconfig level?

no modified .htaccess file, because that should be a patch — Drupal core sometimes modifies its .htaccess file, so you should patch Drupal’s, so you get the latest changes (or maybe the patch will fail if it no longer applies)

no patched contrib modules, that’s what patches are for, and that’s why mr-drupal includes the git_apply_patch function, which you can use like this:

fixups_append = git_apply_patch

no custom modules, because they should live in their own git repository; this repository should only be the glue IMHO

And, indeed, the .mrconfig file can be used to automate many things. Patches are already covered (see above), but permissions are, too! For that, mr-drupal includes the drupal_set_permissions function, which you can use like this:

drupal_set_permissions 'owner' 'group owner'

Another thing that is covered: removing the unnecessary .txt files like CHANGELOG.txt, README.txt, and so on: mr-drupal provides the drupal_rm_unnecessary_txt function for that.

So, maybe a more complete example of the more advanced things you want to do is this one (copied from the .mrconfig file for http://driverpacks.net):

So: file permissions, two core patches, a Google site verification file and a symlink at the root of the site to have nice links for “userbars”. Only the first is highly Drupal-specific. And only the first calls yet more functions — to see how to do that, read the “Advanced: environment-specific actions” section of the article again :)

I love the concept. It is much more development-centric (and space friendly!) than an endless constant (and slow) Ægir/platform/make loop, and I feel that the agnostic approach makes for a more straightforward integration with automated build tools.

I am implementing this model, but considering a few changes for a fully clustered environment with multiple contributors and a git flow-type workflow and would value your input:

The actual mr build will be committed into a “docroot” directory in the mr repository, so the same repository can be used to keep a cluster synchronized. Files will be stored/linked elsewhere.

Custom and third-party modules/libraries/projects will exist in a projects/ subdirectory. Each project in this directory is a remote git subtree. git subtree --squash will be used on third-party repos to minimize noise.

Commits of the builds will be into git flow-style Develop/Master branches for deployment and Develop/Feature/Hotfix-type branches for development.

Commits of the custom projects/modules will be into git flow-style Develop/Master/Feature/etc branches, either by a checkout of the module directly, or via a development pull of the full repo and merged back in.

Features merged back into Develop would ‘trigger’ (manually or on a build server) a mr rebuild/commit of the Develop branch build.

mr would be setup to pull from the local projects/ subdir repositories.

My thoughts are that while this does make the repository larger and more expressive, I am hoping it provides:

What you describe sounds sensible and doable. The mentions of “git flow”, Ægir, git subtree and so on imply to me that you’re working on a large-scale platform.
I personally have no need for those complex setups, but you can definitely can combine mr with them. I’d say: just be careful to not make things overly complex.

I only wonder what you mean by archive and purge at some regular interval for very large projects. Archive & purge what?

As for recent updates or anecdotes: I still use mr + mr-drupal to maintain three websites, and it still is significantly easier than the frustrating process I had to endure whenever I had to use Drush. One is on Drupal 6, two are on Drupal 7.

Finally: please ping me when you have news to share, I’d love to read how you’re using mr-drupal :)

Using subtrees without --squash, commit history from subtrees is included in the main project. If there are many subtrees and you are interested in aggregating the commits in this way, then you might want to manually squash/collapse the history at certain points (yearly) to trim the repository.

Part of the point of mr is that you have a separate git repository for each project. A single Drupal module would correspond to a single git repository. mr can be used to check out (or clone in git parlance) put each such repository in the desired location, without having to import each such repository’s history into a “main” repository.

If you’re going to trim the history yearly, then you will lose valuable historical context when debugging (and particularly when git bisecting).

So either I’m confused by how/where you want to use git subtree, or you’re confused about how mr sidesteps the need for git subtree in the first place. Or maybe I’m just plain confused :)

Right. The difference is that I intend to git subtree the custom modules into the mr repository to consolidate the ‘development environment’ while still maintaining the custom modules in their own repositories. As a developer, you will have the choice of checking out just the module or the entire development snapshot. If you only check out the mr repo, you will have included with it the remote repos as subtrees and can modify/commit/merge/etc as normal. This has the affect of aggregating all related commits into the history of the mr repo, which is desirable from the standpoint of ongoing code reviews/audits for an entire project.

If you only check out the mr repo, you will have included with it the remote repos as subtrees and can modify/commit/merge/etc as normal. This has the affect of aggregating all related commits into the history of the mr repo, which is desirable from the standpoint of ongoing code reviews/audits for an entire project.

I disagree. The mr repo should only refer to tags of actual code repos. The mr repo should only be the glue.

If you’re going to do “actual” development for each Drupal module in its own repository, yet still include the most recent history of each Drupal module using git subtree, then you’re just signing up for a lot of overhead work.

I can see why one might want to be able to have a single repository for all sub projects (i.e. Drupal modules and so on), but I personally don’t need or want it. However, if you want a single repository to rule them all, then the value of using mr/mr-drupal has mostly evaporated. The simplicity I tried to achieve has vanished. Because now it’ll be up to you to manually sync repositories!

For example, imagine you’re working on a foo Drupal module. You’re working on it within the mr repository’s git subtree‘d in version of it… so now you have to manually sync those commits to the foo Drupal module’s own repository. Or vice versa.
Far too large a margin for error, IMO.

It’s much, much simpler to just tag a new version of the foo Drupal module and then update your .mrconfig file to use that instead. Or, if need be, include a patch for foo in your .mrconfig (e.g. if it’s not a Drupal module you own).

Tagging in each subproject’s repository and then updating the mr repository to use those tags. That’s much simpler. That keeps each subproject’s history sane. That ensures the mr repository’s history is both simple and … a nice high-level change log of what has changed in the main project.

Of course, that is just my opinion, and maybe I’m misunderstanding you, but it sounds like you’re making things more instead of less complex :)