Linus Torvalds’s Lessons on Software Development Management

If anyone knows the joys and sorrows of managing software development
projects, it would be Linus Torvalds, creator of the world's most
popular open-source software program: the Linux operating system. For more than 20 years, Torvalds has been directing thousands of developers to improve the open source OS.
He and I sat down to talk about effective techniques in running
large-scale distributed programming teams – and the things that don’t
work, too.
Torvalds
says there are two things that people very commonly get completely
wrong, both at an individual developer level and at companies.
“The first thing is thinking that you can throw things out there and
ask people to help,” when it comes to open-source software development,
he says. “That's not how it works. You make it public, and then you
assume that you'll have to do all the work, and ask people to come up
with suggestions of what you should do, not what they should do. Maybe
they'll start helping eventually, but you should start off with the
assumption that you're going to be the one maintaining it and ready to
do all the work.”
Torvalds continues, “If you start off with some ‘kumba-ya feeling’
where you think people from all the world are going to come together to
make a better world by working together on your project, you probably
won't be going very far.”
“The other thing—and it's kind of related—that people seem to get
wrong is to think that the code they write is what matters,” says
Torvalds. Most software development managers have seen this one. “No,
even if you wrote 100% of the code, and even if you are the best
programmer in the world and will never need any help with the project at
all, the thing that really matters is the users of the code. The code
itself is unimportant; the project is only as useful as people actually find it.”
I'll add at this point that this isn't just a programmer problem.
I've seen entire companies get locked into the idea that “perfecting”
the program was everything. They then neglected what the users wanted
from the program, supporting the users and so on. Most of us who've been
in the business for a while have seen this cycle play out over and over
again.
Expanding on that second point, Torvalds says that's why the Linux
kernel team is “so very anal about the whole ‘no regressions’ thing, for
example. Breaking the user experience in order to ‘fix’ something is a
totally broken concept; you cannot do it. If you break the user
experience, you may feel that you have ‘fixed’ something in the code,
but if you fixed it by breaking the user, you just violated that second
point; you thought the code was more important than the user. Which is
not true.”
Torvalds concludes, “Way too many projects seem to think that the
code is more important than the user, and they break things left and
right, and they don't apologize for it, because they feel that they are
‘fixing’ the code and doing the right thing.”
To that I can only add “Amen!”

On the Importance of Development Tools

I also asked Torvalds about Software Configuration Management (SCM) tools like his own Git version control system. He replied, “I don't think tools are all that fundamentally important.”
“Now, what is important is that there's a good workflow for the project,
and tools can certainly help with that,” said Torvalds. “But most
projects don't necessarily really need tools. There's a lot of projects
that simply don't have enough changes to really require any tools at all
for their work flow; if you only have a few hundred patches per
release, you can maintain those just about any way you want, including
entirely by hand.”
Linux is a different story of course. “For the kernel, we have
thousands of patches flying around every release, and a release roughly
every three months, and so for us the tools really are very important,”
he says. “But I still don't think it was all that big a mistake to just
do tar-balls and patches for the first few years of development; it was a
much smaller project back then, and it took several years for the lack
of tools to really become a problem.”
Besides, “Some tools encourage workflows that are actively detrimental, and I think CVS [Concurrent Versions System,
a version control system] for example has caused a lot of projects to
have the notion of a 'commit cabal,'” Torvalds continues. “I personally
tend to think tar-balls and patches are actually preferable to that – if
only because they make all developers 'equal,' and you don't get the
kind of model where certain people have 'commit access,' and the rest
are second-class citizens. Sometimes it's better that everybody is a
second class citizen than that some people have an easier time at it.”

Torvalds, I should note, knows CVS well and has hated it for years. As he said in a Google Talk in 2007, “I hate CVS with a passion.”

Torvalds continues, “Much more important than the tools is the people. The maintainers, and the mindset.”

Keeping People On Track

And how do these people work together today? I asked Torvalds about the role of the Linux Kernel Mailing List
(LKML) in the process. He replied, “I think Linux used to ‘happen’ more
on LKML than it does these days. The signal-to-noise ratio and just the
pure volume of LKML means that most developers simply don't have time
to really read LKML—at best they scan subject lines. As a result, these
days I'd argue that most of the real development happens within the
sandbox of single developers, and then email on more of a
person-to-person scale is actually how things really get done.”
That said, “That doesn't mean that LKML isn't important; it means
that LKML has become the ‘public band’ of all those individual email
threads,” Torvalds adds. “So what ends up happening is that you have
maybe four or five people involved in a discussion about their work, but
LKML stays cc'd on the whole thing. That turns what would otherwise be a
purely private discussion into something where others can jump in.”
Here's how it works, “A lot of people actually don't really ‘read
LKML;’ they often auto-archive it, but then react to certain keywords
or, more often, [to] key people being involved in the discussion.”
“It also acts as a kind of archiving notion,” Torvalds continues, “so that people can refer to it later, and a lot of bug reports
end up being found by Googling for them. If somebody raises an issue,
it may well be some odd hardware problem, but if Google shows that it's
been raised several times on LKML in the past, that starts to indicate
that it may be obscure, but it's certainly not some totally isolated
issue.”
“So I think LKML is really quite important, but no, it's not how we
keep people ‘on track,’” he says. “All the developers tend to be pretty
self-motivating, and they all have sane ideas (well, the core ones do by
definition – because that's how they became core developers, by showing
that they had good taste and high motivation). It's important simply
because that ‘public part’ of the discussions are still important, even
if in practice it's often a pretty small core in any particular
discussion. Things are simply different when they happen in the open,”
concludes Torvalds.

On Delegating – and Staying Sane

Once, Linux was a solo project. It now has thousands of committers
and contributors. I then asked, “How much delegating do you these days?
Any thoughts on how to delegate to keep one's sanity and the workflow flowing?”
“If there's one thing I've learnt, it is that you have to learn to
let go and not try to control people and the code,” he says. “If you
don't think somebody else can do it on their own without your oversight,
you might as well give up immediately as a maintainer.”
He continues, “Yes, I often get involved in small details, but it's
not because I don't trust people or don't delegate. It's because some
small detail ends up being brought up to me. Either it's a bug (and they
are almost all just silly small details that got overlooked), or it's
just some workflow issue that bothers me (like me complaining about the
developer names not showing up properly in the logs earlier today to one
sub-maintainer).”
Still, says Torvalds, “Those details have to be occasional details,
not the kind of ‘look over the shoulder of the developer to check
everything he does.’ I trust sub-maintainers to do the right thing 99%
of the time. And then, very occasionally, I end up complaining loudly
about something.” Say, for example, on how the open-source GNOME desktop is, or rather isn't, moving forward.
So, there you have it. That's some of the ways Torvalds does it. And,
if you think you know better, ask yourself: Have I created a
world-class operating system that runs most supercomputers,
stock-exchanges, and websites like Google? If your answer's no, I'd
re-read his answers and take a long hard think about how you've been
managing your own projects.