Main menu

We're not an underground success anymore

by
Larry Garfield

Filmmaker John Waters began his career as an "underground success", making films that mainstream wouldn't touch because they were too controversial, raunchy, or just plain weird. After several underground successes, however, he started to get noticed outside of underground and indie circles and move into the mainstream, where he had even more commercial success.

Many of his fans derided him for the move, claiming he was "caving into the man" or betraying his underground roots by adapting his style for wider audiences. In an interview for his 1998 film Pecker, Waters made the comment (sadly I do not have the direct quote handy) that the problem with being an underground success is that you then, by definition, cannot stay underground. And yes, that means you sometimes have to change things.

Why do I bring this up? Because Drupal has long since ceased being an underground success and is now a major mainstream open source player... and it's time that the community acted like it.

Should we go mainstream?

Recently, I've had more than one instance where someone either asks whether Drupal is a hobby project, or seems to treat it as such. To be sure, Drupal had its origins as a hobby side project, and those origins are still apparent in its code base, development workflow, architecture, and in the tools used to build and support it. However, that is not what Drupal is now.

In terms of number of contributors per lines of code in the core system, I am fairly sure that Drupal 6 has more contributors per line that the Linux kernel does.

Drupalcamp Chicago 2009 had over 300 attendees, making it better attended than Drupalcon Sunnyvale (my first DrupalCon). In planning for Drupalcamp Chicago 2010, I don't think 500+ attendees is unreasonable. And we're one of the newer, smaller Drupalcamp communities. New York, LA, and the Bay Area have been around longer and are even larger.

Drupal went mainstream a long time ago. I would argue, in fact, that the turning point was the founding of the Drupal Association; the original impetus for which was that enough third parties wanted to support "Drupal" in non-code ways (donating money, donating servers, etc.) but there was no "Drupal" other than Dries Buytaert himself, which didn't work so well, especially when he was a Ph.D student. And that was in 2007. It's now 2010.

To be sure, Drupal is a hobby for a lot of people. Many of those Drupal hobbyists go on to become major core contributors (including yours truly). Many stay as casual contributors or fanboys, and that's fine. But that's different from Drupal itself being a hobby project. It's not. It's a professional, enterprise-grade system with a large commercial ecosystem backing it, and hundreds or thousands of people who depend on it for their livelihood.

Drupal is growing up into a major professional project. Some of that process is painful, and it changes the dynamic of what Drupal is and how we work with it. But for better or worse, it is inevitable.

To be blunt, those asking if Drupal is a hobby project are at least three years behind the times. In Internet time, that's an eternity.

What does it mean to be mainstream?

Dries has already observed some of what it means to become a mainstream success: Acknowledging that we are not our own primary user base is a big one. But it also means acknowledging what we're not good at, and deciding if we're really going to invest the time (and money) to get good at it or if we're going to seek outside help. It means accepting that some of the jokes you make in an underground movie theater just don't play in Peoria, and some of the architectural decisions you make when your main audience is weekend-PHP-hackers running a blog are different than the ones you make when your main audience is professional site developers and institutional content editors.

What have we done?

In some ways we're already adapting to being a professional project.

The Drupal Association exists in the first place.

When it came time to redesign Drupal.org, we hired professionals rather than relying on limited in-house expertise or casual volunteer work.

The Drupal Association has recently moved to partnering with professional events management firms to run DrupalCon, rather than expecting an infinite supply of unskilled volunteers to manage ever larger events at their employer's considerable expense.

Oh yeah, like what?

As we continue to adapt to being a professional-grade project, not a hobby project, there's a lot of other hard questions we, as the Drupal community, should be asking ourselves as well. For example...

Why do we still use CVS for version control? Yeah, yeah, I know why. It's a cop-out. CVS is holding us back. Full stop. A lot of very interesting and important development is now happening on GitHub or similar third-party sites, and then eventually (we hope) coming back onto Drupal.org in huge untrackable chunks. Whether it's git, bzr, or whatever, we need to do something. Sitting around waiting for "the community to provide" is not working.

In fact, why are we still hosting our own repository in the first place? Projects with far more resources that we do are migrating to hosted code management such as Gitorious so that they don't have to deal with it. We should be open to that as well, so that we aren't wasting time having volunteers fight with a decades-old system. (We should be paying a great deal of attention to KDE, actually, as they're a very similar project, socially.)

For that matter, why are we still spending time on project.module at all? There are other existing tools that offer much more flexibility, and some are even open source.

If we want to continue to spend resources on maintaining our own home-grown infrastructure, we need to commit to spending those resources. That means time and money, and in some cases a lot of it. Right now, we're not doing that.

The Drupal community hired professionals to design the new Drupal.org, but not to implement it. For that, we relied on "community volunteers". While much progress has been made, we're still running the same tired old design over a year later, and still trying to find unpaid volunteers. We're about to release Drupal 7, which will have a new and modern-looking user interface, but there's still no guarantee that Drupal.org won't still be running a design and information architecture that was built for Drupal 4.5 when that happens.

We need to acknowledge that not all man-hours are created equal. Not everything in Drupal can be, or even should be, approachable and understandable by every weekend-PHP-jockey. That's a laudable goal, to be sure, but complex problems sometimes have complex solutions, and we need to accept that. (As complex as necessary, but no more.)

With Drupal being used in more and more mission-critical systems, perhaps it's time to reconsider our long-standing attitude of "we don't babysit broken code" (also known as "if you have a bug, sucks to be you"). It's a great excuse to keep core code lean and mean, but it also means many simple, easy to fix bugs are virtually impossible to find. A robust system needs robust error handling, and "you mistyped something? sucks to be you, hope it doesn't hose your database" is not robust. And yes, that means possibly spending CPU cycles on it.

Most of the highest-profile developers are at least in part sponsored by their employer, but also work a lot of evenings and weekends on their own time. However, Drupal is getting substantially larger, and we cannot rely on volunteer passion and generous employers forever (especially in this economy, and especially when developers have kids, or even grandkids). Will we reach a point where we will have to start paying developers, or at least some of them, just to make sure we have enough skilled hands in the right place? Who can do that, and how would we decide who to pay? I don't know, but it's a question we should be asking.

Maintaining Drupal or a Drupal subsystem is very different than writing code for it. It's closer to management or project management than software development. Should we be looking for more managers and project managers as maintainers? Do we need professional cat-herders, not cats trying to herd other cats?

Software development and software engineering and architecture are decidedly different problem spaces. Should we have people, formally or informally, who are primarily architects, not code slingers? That's going to mean trusting them and their architectural decisions, something that the Drupal community (and open source folk in general) has not always been particularly good at.

Conclusion

To be sure, not all of the above ideas are good ideas. Some wouldn't work, some are just really really stupid when you think about it, some are infeasible for cultural reasons. But we still need to take the time to ask these questions, because whether we like it or not Drupal is not a hobby project anymore and it cannot go back to being one. We're not an underground success anymore, and we're playing in large national theater chains, not just independent art house theaters.

We cannot stop these changes, whether we want to or not. We can either change with them and direct them, and ourselves, into a direction we want, or we can get run over by them.

Larry is Palantir’s Senior Architect and Community Lead. He is an active Drupal core contributor, member of the Drupal Association Advisory Board, the database system maintainer for Drupal 7, and a Drupal 8 Core Initiative leader. Larry is a Zend-certified PHP 5 developer with a Master’s degree in Computer Science from DePaul University.

Comments

I'd say, you got a point. Trying to draw a line between what we want to do ourselves and what not might be a good step. It's a decision every company has to make when it wants to grow. Personally I tend to specialize more and more and leave more and more things aside that are not my core field of knowledge.

It's a decision of where to roll your own makes sense and where not. Instead of hacking on project module we should spend our time to migrate drupal.org to Drupal 7. While I guess we are happy that it runs Drupal 6 now, such a step needs resources.

Especially using github recently shows that there are systems that work well that have nothing to do with drupal. There are many more places where we waste our energy, but a lot of cautiousness is needed to carefully draw the line.

3) Training. Old-school Drupalistas howl as if stabbed when someone proposes Drupal certification or other formal coursework. But its existence validates the product to enterprises. (I personally dislike and distrust such certifications myself... but as you pointed out, I'm not the market.)

4) Staff. Volunteers should remain a central part of the solution, but you need staff to be there consistently and to do the boring things no volunteer will touch.

5) Ways to pay for the staff, infrastructure, etc.. I'm hopeful that commercialization of drupal.COM will help solve this problem.

Some of these things are being handled nicely by Acquia. For example, I didn't mention "marketing" because I think that they're filling in a lot of those gaps, and they've discussed creating a certification program. Some fear the possibility that too much becomes the province of a commercial organization; if you feel that way, the only way to take these tasks back is to take them back.

Most of these considerations make sense for me. Unfortunately, the conclusion, too. ;)

However, to nitpick on something, since I don't have anything else useful to say right now...

I'm not sure why everyone is bashing on Project module and the infrastructure we have on drupal.org. After testing (and often also having to work with) other project + issue tracking "solutions", I really love getting back to drupal.org, every single time. Sure, we should fix subscribing to issues, and perhaps it's also my streamlined view on drupal.org that makes it more usable for me... but in the end, day-to-day things work far better for me on drupal.org than anywhere else.

I agree, actually. I usually describe project.module as the issue tracker that I hate the least. :-) However, it's also a lot of resources needed to maintain and develop it, and there are parts of it that may not be worth doing ourselves. There are existing tools for managing complex repository control, for instance. Why replicate that part ourselves?

Really maintaining project module properly takes more resources than we're currently giving it. The point is that we need to decide to actually commit those resources or not. The current limbo is not sustainable.

Although this post is very well thought through, and will most definitly spark some comments. I think we are still benig to soft on the big problem of resources.

After spending only one-two-year development period, I cannot see how it could ever been done without commitment of certain volunteers who with it lost business value. There needs to be either a push from the community, or from the business deciders - in our community, to put more resources towards Drupal community development. The friday afternoon model, simply doesn't work - lets get serious on this.

It's obvious the community on its own, can not attack this with more and better documentation.

One of the design issues is obviously, the illusion that we don't have the talent inside our community. We do, but we never reward them in anyway - so its hard for them to keep engaged.

@tomgeller 1) Is not a problem, it can take 500 comments - aslong as it gets done "right". We should focus on getting things "right" - its a quality standard we should not shy away from by discussing in length.

You touch on a great point. I have had the opportunity to be apart of numerous back channel conversations where we have talked about architecture and positive changes we could make towards Drupal which have benefits to the community as a whole.

You also make the great point that the friday afternoon model doesn't work. Many Drupalers started off at a different places in their lives than their are now. Priorities have changed. Time challenges are different. And, Drupal has become an enterprise product. Beyond everything else, the culture of the community has changed.

I've seen a lot of people wax poetic about what exactly the je ne sais quoi about Drupal that makes it what it is. The hook system, its strong architectural foundation, its robust and flexible framework...

Let me tell you, it's none of that crap. Hooks are a simple naming convention that could be implemented in any language at all, including BASIC. Most "top-down" managed projects have far more well thought-out and cohesive architecture than we do. There are far, far better frameworks, including those specifically engineered for the task, and who aren't carrying around ~10 years' worth of baggage.

No, the thing that makes Drupal what Drupal is, and is the direct cause of its exponential growth and ascension to greatness is plain and simple: "it's the community, stupid." Drupal (the software) is completely different nowadays than it was back in Drupal 1.0 (or even 4.7, when a lot of us started). There are thousands of add-on modules, where there used to be dozens. The code part of Drupal is wholly unrecognizable now from 2001.

But the stuff that has remained constant (and is the key to both Drupal's past and continued success) is all about our community engineering, not our software engineering. The fundamental values that high-profile members of our community espouse and employ: humility, encouragement, mentorship, collaboration, empowerment. Various processes and tools specifically engineered to help reinforce these core values: peer review, patches against canonical targets, and yes, even that big bad word "centralization." This has all directly led to people getting sucked in and staying hooked to the project, and contributing their asses off, despite the near-universal experience of being frustrated out of your mind when you first try to surmount Drupal's learning curve (more like, cliff :P).

I'm obviously not opposed to us stepping up our game. There are definitely areas where we are not as efficient as we could be, and areas where we're making it harder on ourselves than we should be. Many are identified here. But before we consider some of these radical ideas, the first and foremost question in our mind absolutely has to be "What are the community implications of making this change?" We need to treat anything that has such implications (which, let me tell you, is absolutely everything, from seemingly "no-brainer" infrastructure changes up to more "political" decisions such as whether or not to start paying people) as an amendment to our fricking Constitution in terms of the gravity with which it must be considered. Because we lose that community spirit, and we lose Drupal. Regardless of how much more efficiently we're able to work.

Angie, I am 1000% in agreement with you that it is the community that makes Drupal. And in a sense, that what makes the current questions so challenging, precisely for the reasons you say.

That said, the community has changed in the past and will continue to evolve over time. Such is the nature of communities.

These are challenges that face Drupal, as a community. It may not be PC to ask, but given that Drupal has long since ceased being a blog and become a multi-million dollar industry that runs multiple governments and Fortune 500 companies we must ask ourselves what changes in the community makeup, in the community constitution as you put it, that instigates.

For instance, recently there's been much discussion about whether or not we care about web servers that are not running APC, or other small mom-and-pop sites. We claim to and tried to support that group of users with the D7UX work, but at the same time Drupal 7's system requirements increased dramatically. To address those, we are going to have to make extremely fundamental changes to Drupal's core architecture.

Do we, collectively, care enough about the low-end of the market that we're going to build our architecture around that, or do we, collectively, care primarily about the high-end of the market where everyone runs their own server and build our architecture around that? If we continue to try to say "both, Drupal is for everybody!", what costs do we have to bear in order to try and pull that off? And who is going to pay that cost?

I at this point have little faith that Drupal 7 will run on typical cheap shared hosting at all, to say nothing of successfully. Just how much do we care about that, and what are we willing to sacrifice to solve it?

Drupal.org's CVS repository simply cannot support the number and complexity of modules we're asking it to. Yet we as a community have failed, utterly, to successfully address the problem. We must, or else the community will continue to migrate to github.

Supposedly, much of Drupal's success has been due to its weekend-hacker-friendly architecture. That architecture, though, is reaching (or has perhaps past) the point of diminishing returns. We cannot grow further in that direction. Do we continue to try and maintain that approach, because we value code from HTML jockeys who learned PHP from a book more than software engineers who understand how to solve complex problems in complex ways? Or do we allow parts of Drupal to get rewritten in more long-term sustainable ways, knowing that those parts are therefore less-approachable to people with only limited development experience? And if so, how much of Drupal do we do that to?

Do we try to court more self-taught developers, or do we try to court more trained professional developers? (Hint: You cannot court both at the same time.)

Yes, these are heavy questions. Yes, some of them imply fundamental changes in the way Drupal and the community operate.

That's why it's critical that we ask them openly and directly rather than hoping they sort themselves out. They won't, at least not in a way anyone will like. We as the community must solve these problems, because they will not be solved for us.

Angie, I agree. When I ask, what brought us to these questions and what is going to take us out of them I come to one simple answer. The community.

The community is talking about these topics. More often than not they are back channel conversations. Blog posts like this are bringing them to the fore front so we can move the community forward in a positive direction. At least, that's the goal.

All your questions are important, but if answered "yes", the Drupal Association would become a major employer. The Drupal Association does get a fair amount of resources, but obviously not enough, and I think the first question is:

How do we create a sustainable business/funding model that can provide us with the resources to be able to even address these questions? I am not up on all the open source projects out there, but successful, sustainable models for a community as big as Drupal seem very rare.

Also, accepting the mainstream wholeheartedly runs a big risk of alienating the hobbyists and the numerous people that put in a patch or two. Not to say that it isn't worth the trade off, but those hobbyists are the ones that eventually become the big players. If you create a culture where only employees and employers can significantly contribute to Drupal, you will see a major cultural shift in values, for better or worse.

When we start to ask questions about funding we go down a very political and complicated path. A good place to start with that is to ask tough questions and look at what others have done. For me, two things really stand out.

The first is the tough question. Who will be the next Drupal core maintainers alongside Drupal. This is a difficult question because it's not a matter of who would be great at it but who can put that time commitment in. If you look at Gabor and Angie you see they had companies backing their effort. Would would be great moving forward that has the time?

This becomes more complicated because many of the people who would be truly great as a core maintainer don't have the time to do it on top of their day jobs. When the back channel conversations bring this up the great names fall off for lack of time.

The good example for me is the Linux Kernel development. Linus works for a foundation as the core maintainer. But, a majority of the core developers work for companies with a stake in linux.

What could this look like for Drupal? If Drupal takes the linux route I don't see Dries leaving his startups. He seems to have the startup bug and that's great for him. But, what about the association funding hours for the core maintainers to work?

When I look at the core maintainers being paid to work on Drupal I see it as money to fight some problems we have.

We could get some awesome maintainer to do it as their day job.

The maintainers could put more time into Drupal. There are more Drupal issues that need to be covered than maintainer time to do so.

It draws a clear line of what time is paid for and there is precedent for it.

I have a very split view where it comes to paid work on core development. First a disclaimer, in the past 18 months I've worked on a couple of big projects where at least a portion of the work has involved getting paid to work on improvements to core in specific areas - internationalisation and performance respectively. However, those paid hours only cover a fraction of the work I've done on i18n and performance in core, let alone other patches, so their main affect has been a couple of months where I didn't have to look quite so sheepish when my wife asks me whether I'm working or not.

That kind of funding is still more-or-less indirect - same as many patches to contrib modules, and 'community time' (although I think it's very hard to contribute to core if you only use three full days a month or one afternoon a week - it can take that long to catch up from last time). All of these have been going on for some time, none are going to kill the development process or community, but they don't solve the manpower issue either ("oh I can look at that next Thursday", or "that project is finished now").

Where direct funding of core development worries me is that my main motivation for working on core development has never been to get paid for working on core development (even indirectly), it's mainly been to fix core bugs which affected my first Drupal site (some of them, but not all, are fixed in D7...), and to learn PHP. When I started contributing I barely resembled Larry's 'weekend PHP jockey' persona, more 'weekend CMS + CSS and a bit of copying-and-pasting jockey'. I worry about Larry's apparent dichotomy between luring CS graduates vs. hobbyists far more than his one about dedicated servers vs. shared hosting, Drupal's been in that tricky situation for some time, and it drives a lot of improvements both to the software and the community.

Had there been a handful of people paid full-time to work on core, it's likely I wouldn't have got involved as a regular contributor - since the perceived barriers to getting involved would have been much, much higher. And while a lot of people, including me, work on core in the hope of getting better paid work as a result (skill + reputation = better work and they usually do), that's very different to working on a project for free, when others are getting paid to do the exact same thing. I don't do that out of principle, even though I've turned more freelance work than I'd care to admit so I could keep spending 15-20+ hours a week on D7.

However, I think the exception to the rule for paid core work is the branch maintainer role - since they don't have any control over which patches get written, or how many get thrown at them, or what time core developers ping them on irc for reviews - so it really is like a day job, and a genuine single point of failure (well, two points of failure including Dries but you know what I mean). Not to mention this graph: http://hojtsy.hu/blog/2008-feb-15/drupal-6-maintainers-perspective

For development, I think there could be a lot more indirect funding than their currently is - a lot of improvements to contrib modules get sponsored either directly or indirectly by clients, much less core - due to the release cycle and the kinds of problems it has to solve, but I don't have great ideas on how that can happen.

Yes, well said Tom. Thereâ€™s no doubt that drupal is the geeksâ€™ favourite CMS... Thing is, a lot of managers donâ€™t look on this as a positive. Instead they look at that sinister blue teardrop as an augury of the river of tears that will be spilled as they lose days of their project to a technical infrastructure only fully understood by Vulcans. And whilst I have it on good authority that drupal 7 is going to make my morning coffee for me, the sad fact is that half of the content producers in my company â€“ all highly intelligent â€“ arenâ€™t geeky enough to use the drupal 6 CMS to merely publish an article... Truly, I have nothing against geeks, but in order for drupal to go mainstream, the geeks need to acknowledge that yes, this is actually rocket science and the failure of regular people (muggles, non-magical folk) to grasp its genius is not only reasonable, itâ€™s a serious barrier to adoption. The answer is professionalisation and, as you suggest, this is a major cultural shift.

This is something that I've heard alot from clients and sadly after looking at many of the sites I've seen out there it is usually the case, but doesn't have to be. I've done alot of work with interested clients on the overall content administration experience and the simple changes that we've implemented turn their site into something that isn't even recognisable as Drupal. Too many times when I hear complaints and try to talk through possible improvements, I'm met with unwillingness to spend money on the administrative user interface/user experience. I don't really understand this with clients that are willing to spend thousands of dollars to build a custom site, and then expect an out-of-the-box administrative UI to fit their workflow.

First of all, I completely agree with the main thesis here: Drupal is not a hobby project. It has grown up and continues to evolve to meet the widely different needs of the new audiences that use Drupal in one form or another.

By your own admission almost everything else in this article is debatable. Some are less debatable than others. Some are more debatable.

The single largest concern I personally have is the notion of introducing a financial incentive to core code contribution.

Paying for infrastructure (e.g. server maintenance, performance, scaling etc.) or maybe even code management (e.g. third party project software hosting or productizing the 'project' module and paying to develop that product) are both fine in my books. However, paying for core code is a guaranteed recipe for community disaster.

Cash is not the primary incentive for people to contribute to any project of this type.

Maybe I'm saying this because I'm fresh off of reading "Drive" by Danielle Pink that is all about what motivates people. Turns out money is rarely it - especially not for this type of work.

What motivates people in a community like Drupal is what is already in place and what makes the community great. Sure there is selfish motivation that may lead to money (gaining skill = recognition = higher pay at the day job etc.), but the participation in the community is the DIRECT reward. Everything else is incidental.

Money will not solve all problems, true. And yes, having core development done mostly as a labor of love certainly has its upsides.

But there's far more than just core code. Drupal.org needs lots of work that doesn't end up in core, for instance, as you note.

But remember also the Drupal 6 theme layer was written by someone who was paid to work on Drupal full time. Views and Panels are funded by a major international conglomerate. The D6 and D7 core maintainers were both funded by their employers to do their core work. There's already money in there, and a financial incentive.

How much can we rely on me not sleeping for a few weeks to finish up the home stretch of DBTNG, all on personal time? Or any of the other people who do so? Who's stupid enough to do that twice? :-) (I'm not.)

And if we are going to do so, what does that mean for the quality and consistency of our code base?

I am not necessarily endorsing paying people to write code for core. The question comes up periodically, and I am not sure I would even support that directly. But, for instance, Mark and Leisa got paid to work on core, by Acquia, and it was the largest and most consistent UX-centric work Drupal's ever had. The implementation was left to "the community volunteers will provide..." And the success there has been far more varied.

Maybe it's not money, but a clear path? The roadmap (yep, I'm going to keep using that dirty word)?

"The D6 and D7 core maintainers were both funded by their employers to do their core work. "

No. Gabor was a student through the majority of the Drupal 6 lifecycle, and only started working for a Drupal company around the time of the first release candidate. I might be employed by a Drupal company as my day job, but you'll notice most of my commits are typically made during the hours of 11pm - 3am Eastern. I share the same client load as everyone else at Lullabot.

1. We need a cultural shift in thinking so Drupal can gain more mind share, be more professional and be taken seriously has a product. Drupal is becoming less of the underground, secret handshake, weekend project from the geeks that it was.

2. We need to use professional tools and not reinvent the wheel. Don't be afraid of change and make hard choices for the long term.

3. Drupal development is chaotic, slow and unpredictable. We should formalize and restructure to improve timeliness, predictability and overall confidence with the product.

4. Pay people/design firm to help get the D.O redesign completed. After one year, people are becoming impatient.

5. Move even faster. Get more people involved. Pay more developers who are passionate about Drupal while not compromising the spirit of Drupal and the community.

For #1, I don't have anything to contribute. It's a soft goal with no clear answers.

For #5, I hope Acquia and other sponsors can get even more developers on board to code, review and commit patches. I think if any company whose core income comes from Drupal should contribute back.