Drupal 7: the Webchick behind the Wheel

Katherine Druckman talks to Angela Byron about Drupal 7 and managing a distributed open-source development team.

Angela Byron is one of the most respected contributors in the Open
Source community. She has been recognized for her efforts by Google,
receiving the 2008 Google-O'Reilly Open Source Award for Best
Contributor. She's passionate about the Drupal Project and has
worked tirelessly to ensure that the latest release of the popular
Web platform and CMS is the best version yet. We talked to Angie
about her role as co-maintainer of Drupal 7 and about what it takes
to manage such an important, massive open-source project.

KD: You've had a very busy three years taking the lead as core maintainer
on the newly released 7th version of Drupal. Tell us a little more
about what being the Drupal 7 core maintainer means. What does it entail?

AB: Dries Buytaert is the Drupal Project's founder and project lead, also
known as our Benevolent Dictator for Life. He holds the keys (as in commit
access) to Drupal core. Each new release of Drupal, Dries also appoints one
other person, called a co-maintainer, who is also given commit access and
the responsibility to help set the vision and direction of the release
alongside him.

Neil Drumm was the core co-maintainer in Drupal 5, and he prioritized
usability improvements, including a graphical installer. Gábor Hojtsy was
the core co-maintainer of Drupal 6, and he prioritized numerous
internationalization improvements. In Drupal 7, I was selected. Things I
prioritized included usability, quality assurance, “developer
experience”
(API consistency and other things that make Drupal more enjoyable to work
with), and making Drupal more accessible to designers, themers and people
with disabilities.

“On paper”, a core co-maintainer's responsibility is to review and
commit patches submitted by the core developers. In practice, I found that
80% or so of my time was spent not doing that, but
instead on more “human”
endeavors: identifying people working in similar areas and encouraging
them to work together, helping brainstorm architectural direction on
certain patches, mediating heated arguments, helping new contributors
get involved, promoting major initiatives and so on. So in some ways, it
was a community management role, but with commit access thrown in.

KD: Which part of the three years has been the most intense? I would guess
these past few months, but then I wonder what the pattern has been over the
course of working on this version. How was it at the beginning versus the
middle and end?

AB: Drupal's release cycle goes in three phases:

“code thaw” where we make our wildest dreams come true and add new
features, break APIs and generally pursue world domination.

“code slush” where we focus on polishing the rough edges from code
thaw, API consistency, UI cleanups and performance.

“code freeze” where the APIs get severely locked down, and bug
fixes/stabilization is the name of the game.

The intensity varies, and generally whenever we have a major deadline (for
example, a Drupalcon is coming up or a code freeze date is about to be
declared), things heat up a lot. Most of our major features were introduced
a week or so before one of those major deadlines.

One of the most challenging things was maintaining momentum after code thaw
(aka the “fun” part) completed. People could no longer add their pet
features; instead, the attention turned to “the slog” of bug fixes and
incremental improvements. The number of core contributors died off
significantly as they focused back on their contributed modules or other
projects, and those who were left had to carry the pretty tremendous burden
of
taking us from some 150 critical issues down to zero so we could release.

KD: Drupal has a lot
of core contributors. It's up to around 1,000 now,
isn't it? Obviously a small number of those contribute the bulk of the
code, but it's still an enormous group of people to coordinate. Can
you tell us about that experience?

AB: Yeah, the count from the commit logs was a little less than 1,000, but
that doesn't include people who reported bugs,
tested patches and so on. That number includes a set of maybe
50–75
hard-core people who spend their lives in the Drupal core queue daily and
then an enormously long tail of others who contribute only a handful of
patches. And, bear in mind that every single one of those patches has to be
reviewed and then committed by Dries or me.

It's both incredibly exciting and challenging to work in this environment.
My favourite parts were forming battle plans and kicking around ideas on
how to make Drupal 7 the most amazing release possible, helping new
contributors submit their first patch, seeing the community consciousness
about things like usability, design and accessibility gradually evolve and
accept their importance. The core contributors are amazing, smart,
dedicated and passionate folks who pour their hearts and souls into Drupal,
and I'm honored to work with them every day on IRC and the issue queues.

However, this level of closeness also meant things also could get a bit
personal sometimes. And when they did, it really, really hurt, because
these folks are all my friends, and we have worked so closely together for
years. There definitely were days when I needed to step away from the
computer for a few hours and get some perspective, and even considered
throwing in the towel altogether more than once.

On the whole though, the experience was absolutely amazing. I'm
infinitely proud of the team and what we've managed to accomplish with
Drupal 7.

KD: What can you tell us about the
decision-making processes involved in
developing and maintaining this sort of a project? How do you collectively
decide on a course of action or the best approach to a problem?
Programmers can be married to their ideologies. How do you deal with that?

AB: Drupal is very much a consensus-based community. Problems are
identified, solutions proposed, code written and then discussed among at
least two but possibly dozens or hundreds of people. No change, big or
small, makes it into the upstream code unless it has been marked “reviewed
& tested by the community”, which means at least one other person has
looked at it and given it a nod of approval. This is a great “community
engineering” strategy, because it ensures better quality code and
encourages developers to be civil to each other so they can find
reviewers for their patches.

There's typically very little contention around straightforward bug fixes.
On some of the more esoteric or architecturally facing issues, however,
lots of people start throwing around opinions on approaches, and sometimes
heated ones.

In my role as core maintainer, the best thing I could do in those
situations usually was simply monitor the discussion carefully, but keep
mostly quiet except when things escalated to personal attacks. In
almost all cases, if developers are given free reign to hash things out
among themselves, they're able to come to a mutual resolution without any
intervention. It's important that this happen most of the
time, in order
to build camaraderie and respect among the development team.

Occasionally, however, a stalemate was reached and intervention by a core
maintainer was needed. In this situation, I normally try to take the folks
aside in IRC and see if we can hash out their differences together. Oftentimes a heated discussion on the issue queue dragging on for days can be
settled in minutes when both parties are brought together with a mediator.
I try to summarize the opposing views and explain what is good about each
perspective in an attempt to disarm some of the defensiveness that might
be brought into such a meeting. Neither person's idea is
“wrong”; they both
have pros, but we need to come to the right decision that might be some
mixture of both.

In cases where we weren't able to come to resolution, I'd post my best
attempt at a mutually agreeable solution. This actually very rarely was
the actual solution we went with, but it had a way of
“resetting” the
conversation to be around a new suggestion rather than the old ones, which
usually helped everyone play nicely again—usually.
On the whole, I just tried to remind people that we're all here
to make Drupal better, even that stubborn git who can't see your point of
view yet.

KD: What excites you most about Drupal 7? Do you think Drupal 7
improvements will increase Drupal adoption?

AB: Feature-wise, it's hard to
pick—there were so many things we
added to Drupal 7—but I'd say overall it comes down to the following
things:

Image handling now is included out of the box without the need to download
six or seven additional modules. This is huge for
adoption, at least for
people who want something other than text on their Web sites, who apparently
exist. It also brought with it a host of improvements with our native
file API, which has interesting implications for extending Drupal for use
as a document management system.

The new database abstraction layer has a lot of people really excited.
Both from its support of new features, such as transactions, and support
for more database back ends, but also its new object-oriented syntax.
Basically, everything that ever sucked about our database abstraction layer
has been fixed in Drupal 7.

The new entity and field paradigm is an important shift in the Drupal
Project. In the past, Drupal was very “content-oriented”, and lots of
features were developed that would extend “nodes” or pieces of content in
the system. This resulted in a lot of people trying to shoehorn things
that weren't actually nodes (such as users, comments and so on) into nodes so
they could benefit from these features. In the future, these same
features will be developed as fields, which then can be used across any
entity in the system—users, comments, content, taxonomy terms and
more.

The automated testing framework and more than 30,000 tests we added to Drupal
7 has had profound effects on our community development process. We know
instantly if any patch in the issue queue works or breaks existing tests.
We are free to refactor subsystems knowing we didn't break any of the
existing functionality, and the “test-driven development” mindset is slowly
working its way into our development community's consciousness as a best
practice.

Overall, I'm simply thrilled with the amount of diversity in the core
development team now as compared to the last release. We have an
accessibility team, a usability team, a markup and design team, and a
documentation team dedicated to improving the core in ways that go beyond the
code. My hope is that the leadership shown by early pioneers in these
fields will open the floodgates for new contributors in the Drupal 8
process, and that Drupal 8 improves upon Drupal 7 in all of these areas and
more.

KD: I have played with Drupal 7 quite a
bit, but I can't say I've really
put it to full use yet, and I keep hearing about how much more usable it
is. How much do you think this will affect the Drupal learning curve?

AB: Usability was something that saw tremendous attention during the Drupal
7 release cycle, and we saw a radical transformation in the culture of the
Drupal development community and how seriously this barrier of entry was
treated.

Just after Drupal 6 was released in February 2008, Dries, myself and
several other major contributors went to the University of Minnesota (yes,
Minneapolis in February—that's how much we love Drupal) to perform our
project's first formalized usability study. We were given a room
with one-way glass, tools like eye-tracking software to tell where people
were looking on the screen, and the University found several participants
who had previous Web development experience with tools like WordPress,
Movable Type and Dreamweaver but not with Drupal. In other words, people
in our project's direct target audience.

The results were absolutely shocking and completely transformed the way I
look at Drupal. We found that participants were completely lost as to
whether they were on the front end or back end of their Web sites. They were
unable to find major administrative sections in order to perform basic
tasks. They were mystified by Drupal jargon, and on and on.

A usability team was formed who set about fixing many of the problems
identified in testing, and they fixed a number of important and obvious problems
in the existing UI. Additionally, Acquia funded Mark Boulton and Leisa
Reichelt, a design and usability expert, to take a holistic view of the
Drupal administrative experience and make more wide-sweeping changes, in a
community-driven, collaborative usability experiment called D7UX. Both
initiatives worked in tandem to provide Drupal 7 with a new administration
theme; a set of common administration patterns, such as a toolbar and
contextual links; and a task-based administrative information architecture.
We haven't yet been able to test formally how the usability work we did in
Drupal 7 improved the situation over the results we saw with Drupal 6, but
preliminary feedback from the broader community has been pretty awesome.

This usability work doesn't fully address the dreaded “Drupal learning
cliff”, per se. There still are an awful lot of things you need to
know in order to be a successful Drupal site builder, like what modules you
should use for what and what the heck weird words like
“taxonomy” and
“node” mean. However, Drupal 7 hopefully should require a lot less
customization from site builders to put it in front of their clients and
give them a better leg up on answering the question, “Great. I have a
Drupal site installed...now what?”

KD: It's supposed to be more scalable too. Can you tell us how?

AB: A number of new features in Drupal 7 help with the
situation where your site needs to accommodate huge blitzes of additional
traffic, assuring all visitors of a speedy experience:

Master/slave replication support in the database abstraction layer—database writes are slower than database reads, and reads happen way more
often. A master/slave setup allows you to separate reads from
writes, so that the main database storing all the information isn't
additionally overburdened with read requests, which can be routed instead
to read-only slave databases.

Reverse-proxy support—a reverse proxy, such as Varnish or Pound, can
greatly speed up Web site access by caching copies of pages and then
intercepting requests to serve them, saving the Web server from having to
handle page requests directly. Drupal's settings.php configuration file
now contains directives to enable reverse-proxy support.

Support for content delivery networks (CDNs)—services exist for caching
static files, such as images, CSS and JS, across multiple geographically
distributed computers, which then can be served to visitors more quickly
than a round-trip to the Web server where Drupal is stored. By invoking
hook_file_url_alter(), modules can re-route requests from Drupal's files
directory to services, such as Akamai or Amazon CloudFront.

There's a high-performance distribution of Drupal 6 core called Pressflow
(pressflow.org) from which a lot of these scalability improvements
originated.

KD: For our readers who are more comfortable with WordPress, Joomla or even
platforms like Ruby on Rails and Django, what do they need to know about
Drupal, and in particular Drupal 7, in order to have the best experience
getting started?

AB: For folks coming from other CMSes, such as WordPress or Joomla, the
biggest hurdle to getting started with Drupal is often the “LEGO
block”
approach Drupal takes to building sites with modules. It's common
in other CMSes that if you want to add a photo gallery to your site,
you simply search for a photo gallery extension and choose from a list
of prebuilt all-in-one options.

In Drupal, however, the trend in modules is more toward small, generic,
re-usable components that can be combined and mixed and matched in lots of
various ways. There's not much in the way of off-the-shelf photo gallery
modules for Drupal. Building a photo gallery in Drupal typically involves
creating a content type to provide a data entry form for images, adding an
image field in order to upload photos to the content type, creating a view
of photo images, and so on. Although more elbow grease is required at the
outset, the advantage is that the photo gallery you end up with can behave
exactly as you want. And, the same module that provides an image field for
photo galleries also can be re-used to provide album covers and user
avatars, while the same module that provides a photo gallery view can be
used to create event calendars, RSS feeds and other types of listing
pages. This level of customization and re-usability is what attracts people
to Drupal, but it definitely requires a tinkerer's mindset.

For folks coming from frameworks, such as Django or Ruby on Rails, the main
thing to realize about Drupal is that it's more of a
“framlication” than a
pure framework. Drupal provides ample APIs for dealing with file handling,
session management, internationalization and so on, and it also provides a
“hook” system from which Drupal's base behavior can be extended. However,
it also makes some base assumptions that what you're building is a
Web-based application tracking things like users and content. The advantage
of this is you don't need to recode a new login system every time you
build a site on Drupal; this type of low-level functionality is provided
for you in an extensible way. But, it does mean if you don't agree with some
of the base assumptions Drupal makes, you'll need to spend a bit of effort
developing a module to alter behavior you want to change.
The best advice I probably could give to folks coming from more traditional
Web frameworks is to take the time to explore what Drupal can do without
writing a line of code, which is fairly substantial. Then, learn the
extension mechanisms Drupal provides—hooks, the theme system, the
localization layer and so on—to make customizations in a forward-portable
way.

KD: You've worked on some major, large-scale Drupal projects, and Drupal was
also selected as the platform for Whitehouse.gov. What about Drupal lends
itself to those types of sites?

AB: I think Drupal hits a sweet spot in that it's free, open source and an
extremely capable framework that's constantly evolving. It can be
highly customized to particular use cases, and it has an ever-growing community
with a lot of expertise. Many of the enterprise-level clients we work with
move to Drupal from less-capable, proprietary CMSes that have thousands or
even hundreds of thousands of dollars per year in licensing fees, with bugs
they can't fix themselves because they're beholden to a vendor's schedule.
So the idea of moving to something they can be trained on internally or
hire outside expertise to implement quickly is very appealing.

Sponsored by:

Webinar: 8 Signs You’re Beyond Cron

Scheduling Crontabs With an Enterprise Scheduler
11am CDT, April 29th

Join Linux Journal and Pat Cameron, Director of Automation Technology at HelpSystems, as they discuss the eight primary advantages of moving beyond cron job scheduling. In this webinar, you’ll learn about integrating cron with an enterprise scheduler.