Who rooted kernel.org servers two years ago, how did it happen, and why?

More than two years after unknown hackers gained unfettered access over multiple computers used to maintain and distribute the Linux operating system kernel, officials still haven't released a promised autopsy about what happened.

The compromise, which began no later than August 12, 2011, wasn't detected for at least 16 days, a public e-mail and interviews immediately following the intrusion revealed. During that time, attackers were able to monitor the activities of anyone using the kernel.org servers known as Hera and Odin1, as well as personal computers belonging to senior Linux developer H. Peter Anvin. The self-injecting rootkit known as Phalanx had access to a wealth of sensitive data, possibly including private keys used to sign and decrypt e-mails and remotely log in to servers. A follow-up advisory a few weeks later opened the possibility that still other developers may have fallen prey to the attackers.

For three weeks in September and early October, officials kept kernel.org closed so the servers that run it could be rebuilt. When the site reopened on October 4, a message on the front page prominently warned of the breach and noted the steps taken to rebuild the site. "Thanks to all for your patience and understanding during our outage and please bear with us as we bring up the different kernel.org systems over the next few weeks," the message concluded. "We will be writing up a report on the incident in the future."

Almost two years later, the report has yet to be delivered. The promise to deliver an incident report remained on kernel.org as recently as March 1 of this year, before being quietly pulled the following day. To this day, officials have yet to provide key details, including exactly how many machines were compromised, how the attackers were able to gain root access to them, and what they did once they seized control. The delay contrasts sharply with autopsies that were delivered promptly following twosimilar compromises of Apache.org, the official distributor of the open-source Apache Web server.

"As a user, I think everyone should be a little bit disappointed they didn't execute that transparency in a follow-up," Dan Rosenberg, a senior security researcher at Azimuth Security, told Ars. Without a thorough autopsy, "it's hard to really know what level of negligence was involved in the compromise."

Linux developer and maintainer Greg Kroah-Hartman told Ars that the investigation has yet to be completed and gave no timetable for when a report might be released. He said officials remain confident of preliminary findings that the attackers were not able to tamper with the source code that millions of organizations use to compile their Linux systems.

"We went through many rounds of validation of the kernel releases on the site, regenerating them from the Git tree and old backups," he wrote in an e-mail. "All of them were fine, nothing was found to be tampered with or touched at all."

Git is the name of the system that tracks changes made to the source code for the Linux kernel. It uses a series of 160-bit cryptographic hashes to account for the revisions. Copies of the repository and all changes are then cached in thousands of locations around the world. A mismatched hash in one or more location would quickly indicate unofficial changes. Kroah-Hartman also told Ars kernel.org systems were rebuilt from scratch following the attack. Officials have developed new tools and procedures since then, but he declined to say what they are.

"There will be a report later this year about site [sic] has been engineered, but don't quote me on when it will be released as I am not responsible for it," he wrote.

While there's no evidence that backdoors or other malicious code were surreptitiously inserted, the 2011 breach of one of the world's most important software development organizations should nonetheless remain a concern. That's especially true now that we're living in an era where actors of powerful nation-states have been known to hijack Microsoft's official Windows update mechanism and deliberately weaken cryptographic coding standards. Transparency is more important than ever. If Linux maintainers expect trust, they should make good on their promise to tell users precisely what happened and how.

Promoted Comments

Yep. It's also possible that an underfunded, understaffed org has to prioritize resources and has decided that it is more important to move forward than to figure out exactly what happened. In the ideal world they'd do both, but if you think they're obligated to be idealists... please send kernel.org your resume and number of hours you're prepared to invest figuring out a two-year-old breach that they believe has been totally fixed.

Not everything is a conspiracy, even these days.

Making security a low priority is reason enough to be disappointed and concerned. It doesn't have to be a conspiracy.

The professional response after a long wait would be to say that they still don't have enough data or are unsure, not to just remain silent.

Kind of like not calling your bank to let them know you're going to be 2 weeks late on a payment; It's not respectful.

I can get behind that sentiment.

All we can do is intuit a rationale from kernel.org's silence, which is likely to be inaccurate. I doubt they're hiding anything horrendous, so my intuition leads me to believe that they don't have anything like a solid answer, and nobody wants to put their name on a response that is half guessing and half reading tea leaves.

I agree that something is better than nothing here, and maybe Ars's coverage will shake something loose.

You are utterly missing the point. NOBODY owes you a bleeding thing, nothing. You haven't paid for it, you (probably) haven't contributed time or anything else. How is it you are owed jack all? Linux is just people that write some software.

There are plenty of people who do contribute code to the kernel and who trusted those behind kernel.org to provide a proper, secure service in return. They have been failed and they do deserve an explanation of exactly what went wrong and what steps have been taken to prevent it happening again.

Personally I suspect at this point the root cause is known and it's down to poor security practices by one or more of the "big names" in kernel development. It's the only explanation, short of "they have no idea", that explains the reluctance to reveal their findings. And either of those frankly highlights some very concerning issues with how the kernel team treat security.

During that time, attackers were able to monitor the activities of anyone using the kernel.org servers known as Hera and Odin1, as well as personal computers belonging to senior Linux developer H. Peter Anvin. The self-injecting rootkit known as Phalanx had access to a wealth of sensitive data, possibly including private keys used to sign and decrypt e-mails and remotely log in to servers.

It is ironic that a server named after a greek god was defeated by way of piece of software named after a greek military formation.

On the one hand, I'm now very leery of hacks that would appear to benefit the NSA.

On the other hand, this seems like a very crude attack.

I liked the days when I would briefly consider the government being behind an attack, then dismiss myself for being too paranoid.

Yeah, I think we can safely say that the NSA's hacks and back-doors in kernel.org are professional enough that they haven't been detected, lol. Hell, all they really have to do is ORDER the people involved to put in whatever code they want and not talk about it. People somehow seem to think govts need to be subtle, but it is not so, they have the guns, and the courts, and the money...

No big surprise. I am constantly frustrated by how vague news coverage of security problems can be. We never seem to get a clear explanation about what happened, and there is rarely any followup. I want to know what happened in detail to be able to assess how big a risk something might be.

The promise to deliver an incident report remained on kernel.org as recently as March 1 of this year, before being quietly pulled the following day.

While I certainly can't say I'm comfortable with the report still being absent, the promise was hardly "quietly pulled" - it was part of the "site news" list, and the whole of that was reset in their move of the site to Pelican. Not just the promise.

(Of course, sufficiently paranoid readers may conclude that reset was done partially for just that effect, but I'm not going to wade into whether they're likely to be right, not least because I have no idea.)

A delay this long either means incompetence in the investigation or the whole story is being slowrolled because its worse than we were told.

I'm leaning towards the latter.

It's also possible that they don't have enough information to say with any certainty. If it wasn't detected for 16 days, and they only keep logs for two weeks, there won't be much data to determine exactly what happened.

A delay this long either means incompetence in the investigation or the whole story is being slowrolled because its worse than we were told.

I'm leaning towards the latter.

It's also possible that they don't have enough information to say with any certainty. If it wasn't detected for 16 days, and they only keep logs for two weeks, there won't be much data to determine exactly what happened.

I don't know that's the case here, but it wouldn't be unheard of.

The professional response after a long wait would be to say that they still don't have enough data or are unsure, not to just remain silent.

Kind of like not calling your bank to let them know you're going to be 2 weeks late on a payment; It's not respectful.

A delay this long either means incompetence in the investigation or the whole story is being slowrolled because its worse than we were told.

I'm leaning towards the latter.

It's also possible that they don't have enough information to say with any certainty. If it wasn't detected for 16 days, and they only keep logs for two weeks, there won't be much data to determine exactly what happened.

I don't know that's the case here, but it wouldn't be unheard of.

Yep. It's also possible that an underfunded, understaffed org has to prioritize resources and has decided that it is more important to move forward than to figure out exactly what happened. In the ideal world they'd do both, but if you think they're obligated to be idealists... please send kernel.org your resume and number of hours you're prepared to invest figuring out a two-year-old breach that they believe has been totally fixed.

Yep. It's also possible that an underfunded, understaffed org has to prioritize resources and has decided that it is more important to move forward than to figure out exactly what happened. In the ideal world they'd do both, but if you think they're obligated to be idealists... please send kernel.org your resume and number of hours you're prepared to invest figuring out a two-year-old breach that they believe has been totally fixed.

Not everything is a conspiracy, even these days.

Making security a low priority is reason enough to be disappointed and concerned. It doesn't have to be a conspiracy.

A delay this long either means incompetence in the investigation or the whole story is being slowrolled because its worse than we were told.

I'm leaning towards the latter.

It's also possible that they don't have enough information to say with any certainty. If it wasn't detected for 16 days, and they only keep logs for two weeks, there won't be much data to determine exactly what happened.

I don't know that's the case here, but it wouldn't be unheard of.

Yep. It's also possible that an underfunded, understaffed org has to prioritize resources and has decided that it is more important to move forward than to figure out exactly what happened. In the ideal world they'd do both, but if you think they're obligated to be idealists... please send kernel.org your resume and number of hours you're prepared to invest figuring out a two-year-old breach that they believe has been totally fixed.

Not everything is a conspiracy, even these days.

I completely agree that not everything is a conspiracy. That doesn't mean software developers who expect our trust should be excused from providing an autopsy about a breach that was so potentially serious. How expensive or time-consuming can it be to issue a report like this?

Again, no one is saying they were hacked by a nation state or a criminal gang. Knowing what we know now, though, we certainly can't rule it out, at least until the kernel.org people come clean.

A delay this long either means incompetence in the investigation or the whole story is being slowrolled because its worse than we were told.

I'm leaning towards the latter.

It's also possible that they don't have enough information to say with any certainty. If it wasn't detected for 16 days, and they only keep logs for two weeks, there won't be much data to determine exactly what happened.

I don't know that's the case here, but it wouldn't be unheard of.

Yep. It's also possible that an underfunded, understaffed org has to prioritize resources and has decided that it is more important to move forward than to figure out exactly what happened. In the ideal world they'd do both, but if you think they're obligated to be idealists... please send kernel.org your resume and number of hours you're prepared to invest figuring out a two-year-old breach that they believe has been totally fixed.

Not everything is a conspiracy, even these days.

I completely agree that not everything is a conspiracy. That doesn't mean software developers who expect our trust should be excused from providing an autopsy about a breach that was so potentially serious. How expensive or time-consuming can it be to issue a report like this?

Again, no one is saying they were hacked by a nation state or a criminal gang. Knowing what we know now, though, we certainly can't rule it out, at least until the kernel.org people come clean.

Responsibility? Its a HOBBY, this is a VOLUNTEER ORGANIZATION, not some sort of business. If you want to go in and volunteer to complete this report we'll all appreciate it, but the fact is that any project undertaken by kernel.org or any other such organization only happens because it is interesting enough for someone to take on and do. Alternately some business could pay for it or put some people on it. Looks to me like nobody is volunteering to do it, and nobody has decided to pay for it to be done. Nobody is obligated to do anything, nor has some responsibility to do anything. You can always not use Linux if you don't like the way it is developed and supported. Its as simple as that. Anyone here who's using Linux and isn't willing to contribute to necessary or useful kernel.org projects can basically STFU, eh? Honestly, I'm not trying to start a flame war with anyone, but it bears pointing out now and then that Linux is Linus Torvald's hobby project, not some product you pay for...

You have a responsibility when you run a service, even as a hobby, to take security seriously and follow through on your promises. Their security was compromised for quite a while without them being aware of it, and in the aftermath, as a show of good will they promised an autopsy. They never delivered on this promise, and removed the promise from their website in the clear hope that people would forget about it.

Responsibility? Its a HOBBY, this is a VOLUNTEER ORGANIZATION, not some sort of business. If you want to go in and volunteer to complete this report we'll all appreciate it, but the fact is that any project undertaken by kernel.org or any other such organization only happens because it is interesting enough for someone to take on and do. Alternately some business could pay for it or put some people on it. Looks to me like nobody is volunteering to do it, and nobody has decided to pay for it to be done. Nobody is obligated to do anything, nor has some responsibility to do anything. You can always not use Linux if you don't like the way it is developed and supported. Its as simple as that. Anyone here who's using Linux and isn't willing to contribute to necessary or useful kernel.org projects can basically STFU, eh? Honestly, I'm not trying to start a flame war with anyone, but it bears pointing out now and then that Linux is Linus Torvald's hobby project, not some product you pay for...

If the Kernel team were all amateurs and either working in it in the evening/weekends while maintaining a dayjob or were independently wealthy you'd have a point; but the main people involved are mostly salaried workers whose job is maintaining the kernel. It may have started as Linus's I'm Bored project; but it hasn't been one of those for a long time.

Responsibility? Its a HOBBY, this is a VOLUNTEER ORGANIZATION, not some sort of business. If you want to go in and volunteer to complete this report we'll all appreciate it, but the fact is that any project undertaken by kernel.org or any other such organization only happens because it is interesting enough for someone to take on and do. Alternately some business could pay for it or put some people on it. Looks to me like nobody is volunteering to do it, and nobody has decided to pay for it to be done. Nobody is obligated to do anything, nor has some responsibility to do anything. You can always not use Linux if you don't like the way it is developed and supported. Its as simple as that. Anyone here who's using Linux and isn't willing to contribute to necessary or useful kernel.org projects can basically STFU, eh? Honestly, I'm not trying to start a flame war with anyone, but it bears pointing out now and then that Linux is Linus Torvald's hobby project, not some product you pay for...

I don't think there's any evidence to support the idea that no one is willing to devote their time or expertise to issuing a thorough autopsy. I spent three weeks emailing multiple people inside the organization asking for information. When objections were raised about my initial deadline, I extended it by a week, and then a week after that. The response was the same each time: this is not something the leadership is prepared to move on anytime soon, and no amount of deadline extensions will change that.

So if it wasn't already clear, let me make it explicit now: I am willing to spend as much time as it takes to weigh through all the available data and issue a report on exactly what happened and how. I think a project like that would be highly interesting and worthwhile. Does anyone have a suggestion for how we can go about open-sourcing an autopsy into the causes and effects of this potentially serious breach?

Responsibility? Its a HOBBY, this is a VOLUNTEER ORGANIZATION, not some sort of business. If you want to go in and volunteer to complete this report we'll all appreciate it, but the fact is that any project undertaken by kernel.org or any other such organization only happens because it is interesting enough for someone to take on and do. Alternately some business could pay for it or put some people on it. Looks to me like nobody is volunteering to do it, and nobody has decided to pay for it to be done. Nobody is obligated to do anything, nor has some responsibility to do anything. You can always not use Linux if you don't like the way it is developed and supported. Its as simple as that. Anyone here who's using Linux and isn't willing to contribute to necessary or useful kernel.org projects can basically STFU, eh? Honestly, I'm not trying to start a flame war with anyone, but it bears pointing out now and then that Linux is Linus Torvald's hobby project, not some product you pay for...

If the Kernel team were all amateurs and either working in it in the evening/weekends while maintaining a dayjob or were independently wealthy you'd have a point; but the main people involved are mostly salaried workers whose job is maintaining the kernel. It may have started as Linus's I'm Bored project; but it hasn't been one of those for a long time.

Actually though, that just doesn't hold up IMHO. kernel.org exists as an organization to help its members use Linux more effectively. It doesn't have any obligation to non-contributors at all. Any obligation that its members may choose to feel for each other is simply up to them. If they mutually agree to do something together, and members pay for that thing to happen, and it doesn't happen, then they have some issues with each other. In no case do they owe anyone who isn't an active contributor anything whatsoever, IMHO. Just because Linus is paid by kernel.org for instance only means his job is to work on what the people who are paying for kernel.org and his salary want worked on. If that's a security analysis of a breech then that's what he would be working on for instance. Apparently this report is simply not that interesting to the people with the money, and no volunteers want to do it (or can do it).

I disagree. When you write some code and say to other people "Here you go, have fun with this, do what you want with it" you aren't making anyone use it, you aren't promising them it has some particular quality, and you aren't promising to do ANYTHING with it in the future. You have zero obligation to anyone. If other people don't like your security practices or any other aspect of your project they can go do their own thing. Hell, with GPL they can take YOUR CODE and fork it and do whatever they want with it. If you don't like the way kernel.org works then DON'T USE ITS PRODUCTS! Its as simple as that, and any fantasy you have constructed in your mind that someone owes you something for nothing is your own fantasy and has nothing to do with reality. Sorry.

So by that argument, if I as a hobbyist open up a small roller coaster and allow people to ride it for free, if anything happens that's not my problem either? Hey they just shouldn't have used it?

Responsibility? Its a HOBBY, this is a VOLUNTEER ORGANIZATION, not some sort of business. If you want to go in and volunteer to complete this report we'll all appreciate it, but the fact is that any project undertaken by kernel.org or any other such organization only happens because it is interesting enough for someone to take on and do. Alternately some business could pay for it or put some people on it. Looks to me like nobody is volunteering to do it, and nobody has decided to pay for it to be done. Nobody is obligated to do anything, nor has some responsibility to do anything. You can always not use Linux if you don't like the way it is developed and supported. Its as simple as that. Anyone here who's using Linux and isn't willing to contribute to necessary or useful kernel.org projects can basically STFU, eh? Honestly, I'm not trying to start a flame war with anyone, but it bears pointing out now and then that Linux is Linus Torvald's hobby project, not some product you pay for...

…possibly including private keys used to sign and decrypt e-mails and remotely log in to servers

If private keys were compromised that would definitely be extremely concerning and indicate shoddy practices. For a project as important as this in particular private keys should only be used on networked systems via a smartcard-based solution (Debian project uses the Crypto Stick for example), with the keys themselves generated on an airgapped system. It should not be possible to compromise them even if the dev systems and servers are rooted. They're at the root of a lot of trust and should be defended as such, particularly when doing so isn't expensive or difficult. Hopefully that ends up just being speculation.

If you don't like the way kernel.org works then DON'T USE ITS PRODUCTS! Its as simple as that, and any fantasy you have constructed in your mind that someone owes you something for nothing is your own fantasy and has nothing to do with reality. Sorry.

Oh piss off. "Don't like it don't use it" is a bullshit argument no matter what the situation. Everyone has every right to complain bitterly, campaign about it, lobby others to apply pressure, publicize it, and every other aspect of speech. Of course there is no contractual or legal requirement for the devs to respond but anyone can bring to bear any legal economic or social pressure they wish as well. "No legal responsibility" is important and correct, but is not a "No criticism" card either.

Responsibility? Its a HOBBY, this is a VOLUNTEER ORGANIZATION, not some sort of business. If you want to go in and volunteer to complete this report we'll all appreciate it, but the fact is that any project undertaken by kernel.org or any other such organization only happens because it is interesting enough for someone to take on and do. Alternately some business could pay for it or put some people on it. Looks to me like nobody is volunteering to do it, and nobody has decided to pay for it to be done. Nobody is obligated to do anything, nor has some responsibility to do anything. You can always not use Linux if you don't like the way it is developed and supported. Its as simple as that. Anyone here who's using Linux and isn't willing to contribute to necessary or useful kernel.org projects can basically STFU, eh? Honestly, I'm not trying to start a flame war with anyone, but it bears pointing out now and then that Linux is Linus Torvald's hobby project, not some product you pay for...

I don't think there's any evidence to support the idea that no one is willing to devote their time or expertise to issuing a thorough autopsy. I spent three weeks emailing multiple people inside the organization asking for information. When objections were raised about my initial deadline, I extended it by a week, and then a week after that. The response was the same each time: this is not something the leadership is prepared to move on anytime soon, and no amount of deadline extensions will change that.

So if it wasn't already clear, let me make it explicit now: I am willing to spend as much time as it takes to weigh through all the available data and issue a report on exactly what happened and how. I think a project like that would be highly interesting and worthwhile. Does anyone have a suggestion for how we can go about open-sourcing an autopsy into the causes and effects of this potentially serious breach?

IMHO the underlying issue is someone doesn't want to or can't spend the time and energy to support this. While you may be willing to do the crunching someone would have to gather all the data you'd require and spend probably a lengthy amount of time supporting all the questions you would have about network topology, policies, etc which would need to be answered in order to make the information useful in analyzing the attack. Its not just a matter of someone needs to send you some tarballs and wash their hands of it.

No, I don't have any suggestions. I think at this point there's probably a lot of context and uncategorized information surrounding the basic log files and disk images that has probably been lost in the mists of time and dim human memory. Frankly I doubt at this point a complete analysis is possible any longer. It may not have been very feasible to start with, and that indeed may be the core of the issue. Having participated in these exercises before myself I can tell you from experience that it can be REALLY difficult, basically impossible, to sort out what happened in a poorly maintained infrastructure with bad separations of responsibility, poor log retention, lack of basic security measures, etc. You may well discover details of hacks into specific machines but it can be impossible to sort out the overall sequence of events or know for sure what was and wasn't compromised.

IMHO the underlying issue is someone doesn't want to or can't spend the time and energy to support this. While you may be willing to do the crunching someone would have to gather all the data you'd require and spend probably a lengthy amount of time supporting all the questions you would have about network topology, policies, etc which would need to be answered in order to make the information useful in analyzing the attack. Its not just a matter of someone needs to send you some tarballs and wash their hands of it.

No, I don't have any suggestions. I think at this point there's probably a lot of context and uncategorized information surrounding the basic log files and disk images that has probably been lost in the mists of time and dim human memory. Frankly I doubt at this point a complete analysis is possible any longer. It may not have been very feasible to start with, and that indeed may be the core of the issue. Having participated in these exercises before myself I can tell you from experience that it can be REALLY difficult, basically impossible, to sort out what happened in a poorly maintained infrastructure with bad separations of responsibility, poor log retention, lack of basic security measures, etc. You may well discover details of hacks into specific machines but it can be impossible to sort out the overall sequence of events or know for sure what was and wasn't compromised.

I think this response contradicts your earlier post, specifically that part that says: "Alternately some business could pay for it or put some people on it. Looks to me like nobody is volunteering to do it, and nobody has decided to pay for it to be done."

…possibly including private keys used to sign and decrypt e-mails and remotely log in to servers

If private keys were compromised that would definitely be extremely concerning and indicate shoddy practices. For a project as important as this in particular private keys should only be used on networked systems via a smartcard-based solution (Debian project uses the Crypto Stick for example), with the keys themselves generated on an airgapped system. It should not be possible to compromise them even if the dev systems and servers are rooted. They're at the root of a lot of trust and should be defended as such, particularly when doing so isn't expensive or difficult. Hopefully that ends up just being speculation.

If you don't like the way kernel.org works then DON'T USE ITS PRODUCTS! Its as simple as that, and any fantasy you have constructed in your mind that someone owes you something for nothing is your own fantasy and has nothing to do with reality. Sorry.

Oh piss off. "Don't like it don't use it" is a bullshit argument no matter what the situation. Everyone has every right to complain bitterly, campaign about it, lobby others to apply pressure, publicize it, and every other aspect of speech. Of course there is no contractual or legal requirement for the devs to respond but anyone can bring to bear any legal economic or social pressure they wish as well. "No legal responsibility" is important and correct, but is not a "No criticism" card either.

(Also this is not a volunteer hobby thing, it's business.)

Meh, people can bitch about whatever they want, and other people can tell them to shut up, etc... lol. I don't think the kernel.org people are at all beyond criticism. I think constructive criticism and assistance in achieving a secure solution to their problems is in everyone's best interest. It was only the "I'm entitled to a report and this that and the other" that I was commenting on. Truthfully, a report on the breach would have been a great thing, 18 months ago. At this point it might not really be worth the effort or even possible to produce in a useful fashion. I don't know though, opinions can differ on that.

Maybe they can't determine how the breach happened? This could be the case if the attacker managed to cover his tracks well enough. Though they should really just come out and say it, if that's the case.

I disagree. When you write some code and say to other people "Here you go, have fun with this, do what you want with it" you aren't making anyone use it, you aren't promising them it has some particular quality, and you aren't promising to do ANYTHING with it in the future. You have zero obligation to anyone. If other people don't like your security practices or any other aspect of your project they can go do their own thing. Hell, with GPL they can take YOUR CODE and fork it and do whatever they want with it. If you don't like the way kernel.org works then DON'T USE ITS PRODUCTS! Its as simple as that, and any fantasy you have constructed in your mind that someone owes you something for nothing is your own fantasy and has nothing to do with reality. Sorry.

So by that argument, if I as a hobbyist open up a small roller coaster and allow people to ride it for free, if anything happens that's not my problem either? Hey they just shouldn't have used it?

That would be subject to how law is constructed in your jurisdiction. In some places you could be held responsible for injuries on the basis of the "attractive nuisance" principle. You might also be subject to various regulations, insurance requirements, etc. Software developers are equally subject to the law, however it has been widely recognized that free OSS carries no more obligation than a disavowal of any suitability for a specific use and that in effect when someone uses it they are like a person picking up a bunch of pieces of steel and building their own rollercoaster. They may follow some plans, but its their own device and they're responsible for making it safe. Obviously laws could change on this subject and in the future perhaps Linus Torvalds et al might be held responsible for security problems in Linux, etc. I think that would be a bad idea personally, but there are plenty of bad laws on the books. I'd finally point out that roller coasters and software are quite different and drawing an analogy from one to the other is probably flawed.

You have a responsibility when you run a service, even as a hobby, to take security seriously and follow through on your promises. Their security was compromised for quite a while without them being aware of it, and in the aftermath, as a show of good will they promised an autopsy. They never delivered on this promise, and removed the promise from their website in the clear hope that people would forget about it.

You're absolutely right. And when you consume a free service, you have a responsibility to realize that you are not a key stakeholder. If you want to be treated like an important customer and be able to call people on the carpet for not living up to your expectations, you need to pay them or offer other value.

It is disappointing that kernel.org didn't live up to their promise... but it's also disappointing that so many people who contribute nothing have such an entitled attitude.

Maybe they can't determine how the breach happened? This could be the case if the attacker managed to cover his tracks well enough. Though they should really just come out and say it, if that's the case.

In that case a simple "We were pwned so badly the records needed to figure out what happened were destroyed" statement should have been issued.

On a tangent, kernel.org is high enough profile that if they were willing to give a 3rd party security company access I'm certain they'd be able to find one that would do the work for free and and charge the expenses to its marketing budget.