Apache packages: creating a support vacuum

OSS in the real world

Comment Right at the end of OSCon in September, I got the opportunity to harangue Ubuntu's Mark Shuttleworth about the support vacuum distros such as his create. He didn't have an answer. OK, I was putting him on the spot, but I don't think he'd have done much better if he'd had notice of the question. Frankly, I don't think there is an answer.

So what's the problem? We at Apache know our own software, and are prepared to support it in fora such as the Apache Users mailinglist, and the #apache IRC channel.

Among the diverse issues we have to deal with are the many users who have installed Apache from a third party package. Many of these packages are rearranged so as to be essentially unrecognisable; and in particular, some packagers have very bizarre ideas about how to organise Apache's configuration files. That makes it hard, sometimes impossible, to support the end-users: we may be talking completely at cross-purposes. And for a change, the worst problem is not with Windows packages, nor with commercial Apache packagers like IBM and Oracle (for which, at worst, we can disclaim all responsibility), but with Linux distros. By far the biggest offender, in terms of the support headache it throws on us, is the Debian family, notably Debian itself and its near-clone Ubuntu.

Now, if we at Apache stick to what we know about, we'd tell the questioners to go to a Debian support forum. But the Debian people will tell them to use an Apache forum. Where should the burden of support lie?

newbie: How do I stop it listing all my files when there's no index.html?
niq: Options -Indexes
fajita: Options -Indexes turns off directory indexing. Ask me about mod_autoindex and Options for more details.
newbie: So where do I set that?
niq: httpd.conf
...fajita explains how to find httpd.conf ...newbie: httpd.conf is empty. Do you mean apache2.conf? Or something in conf.d/?
niq: Probably. See your distro's docs.
newbie: huh? it's just a standard apache installation.
niq: distros
fajita: distros sometimes mangle your apache installation, putting apache and its files in different places to the standard. It's hard for us on #apache to help if your installation is mutilated this way.

And no one is satisfied. And that's just a simple example: others involve a lot of confusion with nonsense like, for example, per-module conditional .conf files, and distro-specific ways of manipulating them.

They've configured it right, but Apache ignores it because it needs some extra distro-defined magic to activate the configuration file in question. Now they're banging their head in desperation, and we can't help because we don't know the magic. And we get the blame.

You might of course reply: therein lies a great opportunity; a niche market waiting to be exploited. And yes, there are people and companies (your humble scribe included) who earn our bread and water in something not far from this niche. Though far from being as lucrative as, say, supporting Oracle (with open source and decent documentation, the barrier to entry is far lower), it works. But that's at the top end: medium and large businesses, with custom needs and a budget to meet them. Not all Apache users fall into that category! So, that still leaves a support vacuum.

Back to Ubuntu and my encounter with Mark Shuttleworth. I put the question to him in general terms, citing Apache merely as the example I've had experience with at the sharp end of support. His answer, in similarly general terms, was that Ubuntu would expect to be the first point of support, but he would hope that upstream providers would also get involved. Sorry, but that just doesn't work.

First, Debian/Ubuntu would have to provide a complete set of documentation, at the level of Apache's own. Otherwise, people will naturally come to apache.org, and get confused when it appears to bear little relation to what's installed on their computer. And even if they did provide a full handbook, that still leaves a Tower of Babel amongst third-party articles.

Second, sorry, but speaking as an Apache developer, I make the time to chip in to the task of supporting our users. But I draw the line at taking the time to figure out the quirks of someone else's gratuitously much-mangled package. If there was just the one distro, things might be different, but it's not just Debian/Ubuntu - it's Gentoo, Redhat, SuSE, FreeBSD, Windows, Solaris, etc, etc. If you want me to learn the quirks of your distro so I can support it, you can bloomin' well pay me! And to be fair, the commercial distros do pay Apache people - which may be one reason why the biggest support headache thrown back on us comes from the pure freebies.

Well, OK, so don't support distros then. Is it that hard?

Actually, yes it is, unless we stop supporting anything at all. We can't stop people with distro-induced trouble asking in our fora, and they will blame us (sometimes loudly and obnoxiously) if they don't get what they want. This creates real tension.

Now, there's an experimental effort underway to try to close this gap with documentation. The core of it is a wiki page, now at http://wiki.apache.org/httpd/Recipes/DistrosDefaultLayout. It's driven by Apache support people, frustrated with a constant stream of "huh? what are you talking about?" posts. It's not yet clear how useful this will be in closing the support gap; at the moment, it's essentially a quick-reference card for mapping between distros.

Having sounded off at distro packagers, I should say that they do of course, play a vital role. Packaging Apache, including as they do a lot of third party stuff, lowers the bar to entry for the many users who would be uncomfortable with the idea of compiling a package from source. Some third party packages can be challenging for a non-expert admin to build, and distro packagers can be exactly what they need.

Given all that to contend with, do we really need confusing packages to thrust extra complexities onto both us and our users? A plea to Debian and Ubuntu: if you must repackage Apache in a radically different manner, please document it properly! That includes, along with the manual, a prominent notice to your users, that Apache and third-party documentation and support describe something different, and will not always be applicable!