LCA

Last week I was fortunate to be in Auckland attending linux.conf.au. There are dozens of things I could write about from LCA, inspiring talks and fascinating people doing great things. I haven’t even started watching my backlog of videos from the conference.

The session replicated a 2003 Q&A session with Linus Torvalds, Bdale Garbee and Andrew Tridgell. Rusty Russell had apparently jumped on stage during the 2003 keynote, so this time they invited him up from the beginning. All four of these men are well accomplished and highly respected in the Linux community. I personally have a lot of respect for all of them and the things they’ve achieved.

First Questions

The first audience question was asked by former kernel developer Matthew Garrett:

Over the years, various people including myself have either reduced their involvement in the kernel community or stepped away from it entirely due to the tone on LKML [Linux Kernel Mailing List] and especially your contribution to that. Why do you continue to argue that being really unpleasant to people is a good leadership strategy?

Linus’ answer (video here) was a defence of his leadership strategy. His reply includes some click-bait friendly quotes, such as “I’m not a nice person and I don’t care about you.”. The core of his answer was “I care about the technology and I care about the kernel, and I really think that a lot of projects in the open source community sometimes care about non-technical things too much.” Linus said that although non-technical issues were important, “to me… diversity is not about gender, it’s not about skin colour, it’s about […] People are different in what they’re interested in, people are different in what they’re good at, skin colour and gender and all these issues […] those are details. What is great about open source is that some people are unpleasant but they’re technically really good, some people are pleasant and they like bringing other people in […] you can do what you’re good at.”

He also said “When you look at bringing in minorities, bringing in females, bringing in people who don’t speak English – my argument has been and is that we should look for people who are good at being the people who can be between other people. There are lots of good kernel developers who are great at working with people. They may not even be great technically….”

The distinction between those who are “good at people” and those who are “good at tech” seemed to run through a lot of Linus’ answer.

However there are more concerning things that could be implied by this quote, such as that if the kernel project is going to bring in “other” people then they have to bring in go-betweens first, because there’s no way the established misanthropes in the Linux kernel community are going to be able to deal with them otherwise.

The second audience question was similar to the first. It was asked by Thomi Richards, a software developer:

I don’t work in kernel development, I’m not subscribed to the LKML, but I am in the community and I’ve noticed a worrying trend. Which is that in the last year there’s been more hate, there’s been more abuse, more vitriol. On IRC, on mailing lists, on Google Plus. And speaking as a professional this really concerns me. Because I love my job, and I want to love it in twenty years time. But if this trend continues then I will not be in the industry on twenty years time. … Linus has already answered, but for the others … do you have any plans to try and improve the community?

The answer this time (video here) largely consisted of telling Thomi that things were getting better, and a discussion of the “emotional” debate around systemd. BDale gave an extended answer about systemd and Debian that I thought was very insightful, although it painted the Debian community leaders as waiting quietly for the fuss to die down rather than proactively leading. Despite this I thought the panel’s answers all missed the real essence of what Thomi was asking (Thomi was also unimpressed).

Stalemate

So far this discussion follows some well-worn paths, and I think they lead to something of an stalemate.

From one side, there are those who believe that Linus’ conduct – and those who emulate him – is not conducive to a positive experience for kernel developers and discourages participation. They believe that this is harmful. People who hold this view are often associated with a concern for increased diversity or social justice issues. Even though the questions quoted above didn’t explicitly mention these issues, the answers given on stage did mention them.

From the other side, Linus’ answers centre around his right to be unpleasant and offensive, and points (at least implicitly) to the success of the Linux kernel as an example of why this isn’t a problem for him. There’s also an element of realpolitik here – Linus is the project leader so he gets to explain diversity from his perspective, and assign it the importance he considers appropriate.

Another Question

A short while later this question got asked:

Q: Those of us of a certain age got into computers mostly from the 8-bit computers […] I’m just wondering what your thoughts are on getting the younger generation into our community? […] It doesn’t seem there’s much encouragement for the younger ones to join our community, and fill the ranks as we age.”

Tridge : By younger ones you mean under 30?

Q: Yes, I guess.

Linus’ answer here was mostly about his kids, and the technical complexity of getting into modern kernel programming (video here). “The world has moved on, and we’re old farts for a reason.” Regarding kernels, he said “we don’t need everybody to do that“.

Commmon Thread

In all three of these questions I see a common thread – people (particularly younger people) not wanting to engage with kernel development or the Linux community in general. It’s not even necessarily a diversity issue – Matthew Garrett & Thomi Richards are both younger white men, demographics traditionally over-represented in open source ranks. I’m in that same demographic, and with a background in systems programming and writing hardware-level code I’d be naturally interested in learning to contribute to the kernel. The major detractor for me is the community’s demeanor.

I don’t mean to play down the importance of diversity in open source. I think these issues are also extremely important and I think Thomi and Matthew do as well. It’s just that even if you leave the (traditionally polarising) issue of diversity completely aside, the answers we heard on Friday are still problematic. Considering the diversity angle just compounds the problem with additional layers of alienation.

Meanwhile kernel developers are getting older, senior turnover is slow. The outreach efforts of the Linux Foundation butt up against the head of the project proclaiming that it’s OK for him to be unpleasant, and ignoring people who are prepared to say to his face that he (and the example he sets) are the reason they don’t want to be involved.

I obviously don’t have any evidence that these things are broadly connected, maybe “dissatisfied would-be contributors” like us are statistically insignificant. However I wish at least one of the panel members had acknowledged or rebutted that factor when it was stated in the questions, rather than skipping over it.

Regarding the stalemate, I have no great solutions to offer. Linus is clearly a good technical lead for the kernel, and he also seems comfortable with his role as reluctant leader-celebrity (I saw numerous people getting photos with him at LCA). At the same time he eschews any broader awareness of his non-technical leadership role, beyond self-justification.

Perhaps if the aging kernel developer trend continues then another kernel-level project will eventually overtake while Linux ossifies, but that seems likely to take decades. I don’t really want Linux to fail either, after all it contains person-centuries of effort.

And while younger white male software developers are having their opinions panned by the respected older generation on stage, what does this mean for actual marginalised groups? If FOSS is ever going to achieve broad adoption, it has to appeal to more than a privileged few.

I’ll be back at linux.conf.au next year for sure, but if Linus gives another Keynote then I might just read his kernel changelog and then sleep in that day.

Post navigation

7 thoughts on “The Elephant in the Keynote (LCA 2015)”

Nice write-up. There are a few things I’d like to add, for the record:

First, my decision to ask the question I did was not spur-of-the-moment – rather, I’d been debating with myself for many weeks beforehand as to whether I should ask the question at all, and if I did, what form it should take. I have a huge respect for everyone on the stage, and I don’t want any of them to feel unwelcome, or to feel like I’m not grateful for their contributions.

While I was unimpressed with the reply I got, I wasn’t surprised. You have to remember that these gentlemen are all hugely successful uber-geeks – they have a network of people to work with, and haven’t had to try and “break in” to a development circle for years. They simply don’t see how the abuse and negativity turns contributors off, because, from their perspective, they never even see those people.

I said in my question that I’m not a kernel developer – that’s only somewhat true: I have written kernel device drivers before (although closed source ones, for hardware that’s not possible for consumers to purchase), but there is simply no way I’m going to even try to contribute to the kernel while LKML remains the toxic place it is today. I stopped subscribing to that mailing list a while ago, and I’m given to understand that it hasn’t changed.

Another data point that was mentioned was systemd. Lennart Poettering recieved death threats. Actual honest-to-god death threats. Let’s stop a moment and think about that. This guy developed software that will manage the daemons and boot sequence for debian and it’s derivatives (including Ubuntu). His reward for all that hard work? Death threats. At the same time that’s happening, we have technical leaders saying that there’s not a problem? “Unimpressed” does not even begin to cover my reaction. Where were the prominent technical leaders saying “death threads are NOT OK”? Where were the technical leaders trying to moderate the debate? Instead, we have Bdale talking about people’s “emotions flowing over” – that’s fine, but one day someone will get seriously hurt, or worse, die, due to these “overflowing emotions”.

Someone on the panel mentioned that they thought the negativity I’ve seen are part of a vocal minority. They may well be right, but I think that’s beside the point. The fact that these people are a minority in the community doesn’t lessen the damage they can cause. Since I asked the question, I have spoken with developers who have suffered from clinical depression largely due to the negativity they face in their day-to-day jobs. The vocal minority are doing very real damage.

Despite all that, it’s important to remember that the only person you can change is yourself. I work for Canonical, I hack on Ubuntu in many open source projects, and I try as much as I can to conduct myself in a friendly, professional manner. I’m sure I’m not a role model for anyone out there (absolutely no one would ever describe me as an “uber geek”), but in case that does one day happen, I’d like to be a good role model. I can’t force anyone on that stage to change their behavior, all I can do is ask, politely and respectfully if they have any plans to help us change the community for the better. I’m sick of being in the silent majority, so I’m going to start being less silent. I’m trying to tread the fine line between being vocal, but remaining respectful (the LAST thing I want is to emulate the behavior of the people I’m asking to change).

My hope is that, from now on, at EVERY technical conference, wherever community / technical leaders are on stage, someone from the audience steps up and asks, politely and respectfully “what are your plans to help improve the community?”.

Thanks for your blog post, and thanks for reading this far – sorry if it’s a bit long :D

If you pay any attention to the growing quantity and quality of open source projects on Github you will see that this is not nearly the problem you describe, the kids are engaged. But even more interesting – it has always been thus. The Internet has been a miraculous innovation creating astonishing value all over the planet. And it has been one ugly place since 1991 when I first logged on. While it may not be a happier friendlier place today – it is no worse.
And complaining about others is no way to make it better. Don’t waste your time suggesting improvements for others to make, as Linus pointed out: “we don’t care”. Be the change you’d like to see.

Thanks for commenting, I appreciate you taking the time to read my blog post.

I think you maybe misunderstood the “kids aren’t engaged” problem I was talking about. In the post I’m really only talking about the Linux kernel in particular, not FOSS in general.

You say that the internet is no worse in 1991 than it is now, but I think there is a lot of data that disagrees. People, mostly women, in our community have recently been victim to “SWAT-ting” – calling a SWAT team to someone’s house in an effort to have them killed or terrified into silence. Was that part of your experience of the internet in the early to mid 90s? FOSS developers got death threats last year, how prevalent were those at that time? I can provide other examples if necessary.

(Also – if you’re right and the internet has always been this bad – is that really a reason to accept it?)

In the LCA keynote you gave (that I attended), you used the phrase “meritocracy” several times to describe the FOSS community. Would you describe kernel development as a meritocracy? If there are people – like the ones mentioned in this post – who are qualified to potentially contribute, but are self-selecting themselves out of the process because they don’t want to be in an abusive working environment, would you still describe this as a meritocracy?

Regarding “be the change you’d like to see”, I’ll continue to do my best in my contributions. I disagree with you when you say that we shouldn’t suggest improvements that others could make – even if they “don’t care”. I support Thomi’s excellent comment, posted above yours, where he suggests that those of us in this community who care about inclusiveness keep the discussion open and keep asking our leaders politely about what they’re doing to make the community better.

A phrase springs to mind that was popularised here in Australia by an established older leader in a traditionally male dominated field – the standard you walk past is the standard you accept. https://www.youtube.com/watch?v=QaqpoeVgr8U

One issue for kernel development could be a decline in system level programming in the Open Source field in general. Toy kernels used to be a much bigger scene than it is now – partly because of Linux, for sure, but also to a large degree because the interesting things happen elsewhere and ‘kernels are a solved problem’.

As for ‘the standard you walk past’, also consider that the message certainly came across. The people who complained certainly made sure not to walk past. If Linus doesn’t want to change (for whatever reason), what is the next escalation level – evicting him from his own project? He never forced anyone to work with him. I mean, realistically speaking, not some “we’ll annoy him until he turns ‘nice'” fantasy.

One solution could be to build an alternative community. Feed in the code regularly (since the technical merits aren’t in question), but keep the undesired conduct out. Somewhat akin to eglibc routing around Drepper.

That’s actionable and demonstrates what such a community will look like. And most of all, it doesn’t require anyone else to change.

I don’t know where you got the idea that I was arguing for the status quo, or that I was suggesting you walk past broken standards without trying to do something about them. I was just arguing against complaining. Which I suppose could be described as complaining about complaining.

The problem with complaining about Torvalds’ leadership of the kernel, is he has literally made that free software project his life’s work. His style may be tough-love, but it is love nonetheless.

None of us are close to perfect so by definition there are going to be lots of things to complain about in his leadership of the kernel development if you go looking for them. But the folks complaining about his leadership don’t seem to be willing to acknowledge while far from perfect Torvalds leadership has been very, very good as measured by the success of the Linux kernel project.

Try to rally folks to contribute to a positive action, whether developing tools to reduce the influence of the crazies on the ‘net, or contributing to the Hurd kernel project if you aren’t willing to tolerate the Linux Kernel development culture. But complaining is not helpful.

If running a meritocracy conflicts with white knighting fir SJW snowflakes than I’m going to support a meritocracy . I’ve read this article and Bobs and they both raise all kinds of ambiguous and nebulous feel good jargon that no right minded folk could possibly object to. But this is all propaganda for the basic desire to make things “fair” for less talented developers because of their skin color or genitalia.