HBGary’s open letter: full of denials that don’t hold water

HBGary has finally broken its silence about the Anonymous attacks and their …

HBGary, the security firm that saw its servers hacked and its e-mails released after its HBGary Federal offshoot angered the Anonymous hive, published a rather peculiar open letter this past Friday in an effort to address the "large amount of misinformation reported in the press." But the letter makes some questionable claims of its own.

The unsigned letter outlines the basics of the attack and asserts that HBGary's internal systems remained safe and uncompromised. To ward off future attacks, the letter also claimed that HBGary's website, which was hacked using a basic security flaw, and its e-mail system, which fell victim to weak, re-used passwords, were now back in operation with "even stronger cyber defense mechanisms."

HBGary says that the company's concern in the immediate aftermath was to determine if customers had been affected by the intrusion. On receipt of legal advice, the company's policy was to refrain from commenting on the e-mails, though it acknowledges that this may have led to the amount of "misinformation" floating around.

Deny everything

The main thrust of the letter is an effort to distance HBGary from the entire hack and its subsequent aftermath. Five specific claims are made: that HBGary and HBGary Federal are distinct, with separate "management, employees, and missions"; that HBGary was not involved in the research performed by then-HBGary Federal CEO Aaron Barr and was merely caught in the crossfire; that HBGary did not develop Stuxnet; that HBGary does indeed sell software to the US government and is proud of that fact; and finally, that HBGary's rootkit research is solely to help improve its own security products.

While the claims about Stuxnet and software sales to the US government are uncontentious, the others are more than a little surprising. For a start, some of the claims appear to be contradicted by the extensive e-mail dumps. Though HBGary representatives have implied that some of the e-mails may have been tampered with, the prodigious quantity of mail precludes any substantial effort to create fraudulent mail (and the company never responded to our request to identify any instances of such fraud).

While HBGary Federal was legally a distinct company (albeit one with some overlap in ownership), both the hacking methodology and e-mails subsequently published make clear that this distinction was far less clear in practice.

The hack itself revealed that HBGary and HBGary Federal used a single Google Apps account for its e-mail. Former HBGary Federal CEO Aaron Barr, whose actions provoked the hack in the first place and whose password was cracked, had administrative access to both HBGary and HBGary Federal mails. The e-mail accounts of HBGary Federal employees used the hbgary.com domain, not hbgaryfederal.com. HBGary Federal COO Ted Vera had access to a Linux server used by HBGary for providing support to its customers. And the e-mails themselves show that Aaron Barr was in regular correspondence with HBGary CEO Greg Hoglund. The two also worked together to decide how best to word press releases to promote HBGary Federal's work to uncover Anonymous.

Indeed, from day one, the lack of separation between the companies was clear. Greg Hoglund's e-mail introducing new hires Aaron Barr and Ted Vera had the subject "Welcome Aaron Barr and Ted Very to the HBGary management team!"—hardly supporting the open letter's claim of "completely different management."

From: Greg Hoglund <greg@hbgary.com>
To: all@hbgary.com
Subject: Welcome Aaron Barr and Ted Vera to the HBGary management team!
Date: 2009-11-23
I am extremely excited to announce that Aaron Barr and Ted Vera have joined
the HBGary team! Ted and Aaron will operate and lead HBGary Federal, a
wholly owned subsidiary of HBGary, with a focus on contracting in the
government space. They are very experienced and most recently built a
$10 million/year business at Northrop Grumman. Both have won and lead
multi-million dollar development projects and managed substantial teams.
We have known Aaron and Ted for more than 5 years. These two are A+ players
in the DoD contracting space and are able to “walk the halls” in customer
spaces. Some very big players made offers to Ted and Aaron last week, and
instead they chose HBGary. This reflects extremely well on our company.
"A" players attract "A" players. Aaron will take position as CEO of HBGary
Federal, and will be operating out of the DC area. Ted will take position as
President and COO of HBGary Federal, and will be operating out of Colorado
Springs. Welcome aboard!
-Greg Hoglund
CEO, HBGary, Inc.

So while the companies were legally distinct, their management and employees plainly, in practice, were not "completely different." At the very least, HBGary senior management oversaw and were involved with HBGary Federal's operations, and HBGary Federal employees had access to HBGary systems.

Next, the letter specifically distanced HBGary from Aaron Barr's research. The investigation into Anonymous was entirely HBGary Federal's doing, it says, and HBGary was an innocent bystander, caught in the crossfire when Anonymous sought retribution. Prior to the entire issue blowing up spectacularly in their faces, however, nobody from HBGary wanted to be distanced from the research.

From: Aaron Barr <aaron@hbgary.com>
To: Penny Leavy <penny@hbgary.com>, Greg Hoglund <greg@hbgary.com>
cc: Ted Vera <ted@hbgary.com>
Subject: BSides Talk
Date: 2011-01-22
Hey Guys,
I wanted to inform you of my research and content for the talk at Bsides.
I have focused some of my research and talk around the anonymous group, a
supposed loose collection of freedom of speech enthusiasts, anarchists, etc.
They used to target the RIAA with DDOS attacks now they have taken up the
cause of wikileaks, tunisia, venezuela, algeria, etc. They have received
a decent amount of press about this.
I am enumerating their communications infrastructure and plan to brief this
as well as outing many of the major players within the group. This will
likely make HBGary Federal, and likely HBGary a target.
I have developed a persona that is well accepted within their groups and want
to use this and my real persona against eachother to build up press for the
talk. Pre-talk plan.
I am going to tell a few key leaders under my persona, that I have been given
information that a so called cyber security expert named Aaron Barr will be
briefing the power of social media analysis and as part of the talk with be
dissecting the Anonymous group as well as some critical infrastructure and
government organizations
I will prepare a press sheet for Karen to give to Darkreading a few days
after I tell these folks under persona to legitimize the accusation. This
will generate a big discussion in Anonymous chat channels, which are attended
by the press. This will then generate press about the talk, hopefully
driving more people and more business to us.
But it will also make us a target.
Thoughts?
Aaron
--------------------------------------
From: Greg Hoglund <greg@hbgary.com>
To: Aaron Barr <aaron@hbgary.com>
Subject: Re: BSides Talk
Date: 2011-01-23
Well,
I don't really want to get DDOS'd, so assuming we do get DDOS'd then
what? How do we make lemonade from that?
-Greg

HBGary may not have been aware of the full extent of Aaron Barr's investigation, but completely in the dark they were not; Aaron Barr outlined his research to Greg Hoglund and Penny Leavy, Hoglund's wife and co-owner, on January 22nd, two weeks before the eventual hack. Though Hoglund expressed concern that the research might result in a denial-of-service attack against the company, he showed no qualms about either the subject matter of the research or Barr's investigative methods. When Barr's research started to get some publicity—notably, a story in the Financial Times—Hoglund was quick to praise Barr and leverage the media coverage to promote the companies.

HBGary may not have known everything that Aaron Barr did, but HBGary Federal was plainly operating with the backing of the parent corporation.

Ethical concerns

Perhaps more significantly, however, the open letter doesn't really distance the company from some of the more alarming or damaging revelations. The letter admits that HBGary sells software to the US government, and performs some amount of in-house rootkit development, but leaves the more substantial claims unaddressed. The e-mail trail showed that HBGary was pitching its rootkits to defense contractors, and writing what can only be described as malware; hostile programs that would exploit security flaws and install rootkits.

The 12 Monkeys rootkit

The letter claimed that this malware had never been used to attack "foreign countries," and that HBGary knew of no instance of production deployment. Instead, these novel rootkits were intended merely to "understand the offensive nature of our foes." Unfortunately, that doesn't really make much sense as an excuse. The company boasted to potential buyers of its rootkits how they went undetected by standard anti-malware software, a feature only useful if the software is going to be used in the wild. Greg Hoglund, in describing the plans for the HBGary Magenta rootkit, made clear that, in his view, nothing like it existed; this was no mere copycat of existing in-the-wild rootkits, but something new and unique. If Hoglund's assessment is accurate, the insight such a rootkit would give into HBGary's "foes" is negligible—it was technology that those foes hadn't invented and weren't using. This would be useful for attacking, but much less useful for defending.

HBGary's Magenta project

HBGary may very well not actually know about real deployments, but that's missing the point. The concern over its actions were not that the company had explicit, detailed knowledge of actual hacks using its tools—of course the government wouldn't tell HBGary if this were the case. Rather, they concerns were that the company was developing these undetectable rootkits, selling (or at least, attempting to sell) them to defense contractors, at which point they could be sold to essentially any agency for any purpose—they could just as well be used to spy on domestic dissidents as they could on foreign powers.

From: Bob Slapnik <bob@hbgary.com>
To: Greg Hoglund <greg@hbgary.com>, Penny C. Hoglund <penny@hbgary.com>
Subject: Need 12 Monkeys price
Date: 2009-04-14
Greg,
Ben [redacted]'s customer may have $200k to spend. Ben wants us to give him
a *price* to sell the software as a license where we retain the IP. The work
would be firm fixed price (not hourly). This will not be viewed as contract
work -- it will be viewed as our selling a product.
We should only take the work if we believe we can succeed and do it in 2
months. It looks like, if awarded, our work will start in about 2 months.
12 Monkeys Details. Ben needs price quote for a complete tool that
- finds MS Office files using the XRK technique to exfiltrate files
- uses the 12 Monkey technique to exist and hide.
- Runs on MS Windows XP sp2 and Office 2003.
- both Client and Server side.
Ben wants us to retain the IP so he can sell it to more customers.

Concluding the letter, HBGary bemoans the state of the press, complaining of low standards of fact-checking and a failure to verify information—something that it blames on the "blog-o-sphere." This comes after admitting earlier in the letter that the company's own refusal to communicate with the press (something that Ars Technica experienced firsthand) was likely to blame in part for that incorrect information. Though some of the speculation surrounding the case was indeed wild (see: Stuxnet), there was little reason to make things up; the facts alone were remarkable enough.

The open letter is a strange thing indeed. If HBGary's aim is to rehabilitate its image, flat denials of the facts revealed in the airing of its dirty laundry are unlikely to be effective. Claiming total ignorance and blaming Aaron Barr and HBGary Federal for everything—in contradiction of the e-mail evidence—is unconvincing, and failing to even acknowledge the serious ethical concerns about the way the business operated means that question marks over the company's conduct remain. But perhaps the letter simply reflects a corporate mindset that ethical constraints are irrelevant so long as one doesn't get caught. On that front, the letter is quite reassuring.

Nowhere is the old adage "Better to keep quiet and be thought a fool than to speak and remove all doubt" more relevant than big-time public relations. HBGary would have done themselves much better by ignoring everything related to this fiasco, keeping their heads down, and waiting for everyone to forget. It won't be that long, if it hadn't happened already for the vast majority of the public. Releasing a statement at this point is basically a fool's errand to begin with, releasing *that* statement will at best just remind people of the incident and at worst will do further damage on its own.

I have no idea what they hoped to accomplish from releasing any public statement rather than just dealing with the stakeholders and concerned clients. In private, there's much more leeway to just handwave away issues, make empty promises, and dance around the issue. The people you're talking to also have a vested interest, so they're much less likely (sad, but true) to call you out on it. A public statement is just asking for articles like this.

Nobody at HBGary, Federal or not, is looking like a winner based on anything they've done before or after Barr's shenanigans.

As for the actual article, between this and the Rightshaven coverage, it's beginning to seem like Ars is creating "bad guys" and running every bit of bad news about them as possible. The readers join in the hatred of the "bad guys", and boom, page views shoot up as people show up to see the "bad guys" lose, or read about their dastardly deeds and get hacked off. (Reminds me of the political blowhards, which have been using this model for ages. ex: Rush, Bill O'Reilly, etc.)

To be clear, I'm not defending Rightshaven or these guys in any way, but this kind of coverage begins to get cheap and pulpy. (Really, a press release full of BS? Stop the presses!)

As for the actual article, between this and the Rightshaven coverage, it's beginning to seem like Ars is creating "bad guys" and running every bit of bad news about them as possible. The readers join in the hatred of the "bad guys", and boom, page views shoot up as people show up to see the "bad guys" lose, or read about their dastardly deeds and get hacked off. (Reminds me of the political blowhards, which have been using this model for ages. ex: Rush, Bill O'Reilly, etc.)

To be clear, I'm not defending Rightshaven or these guys in any way, but this kind of coverage begins to get cheap and pulpy. (Really, a press release full of BS? Stop the presses!)

Except this is totally deserved. Their "open letter" is proof enough that they just won't learn, and are in fact, "bad guys".

As for the actual article, between this and the Rightshaven coverage, it's beginning to seem like Ars is creating "bad guys" and running every bit of bad news about them as possible. The readers join in the hatred of the "bad guys", and boom, page views shoot up as people show up to see the "bad guys" lose, or read about their dastardly deeds and get hacked off. (Reminds me of the political blowhards, which have been using this model for ages. ex: Rush, Bill O'Reilly, etc.)

To be clear, I'm not defending Rightshaven or these guys in any way, but this kind of coverage begins to get cheap and pulpy. (Really, a press release full of BS? Stop the presses!)

Ars did a massive amount of coverage of the HBGary Federal issue, as we all know. Following up on that in-depth coverage with a reasonably short piece centering on the company's official response isn't "cheap and pulpy." It's actually the exact opposite; it's good reporting.

As for the actual article, between this and the Rightshaven coverage, it's beginning to seem like Ars is creating "bad guys" and running every bit of bad news about them as possible. The readers join in the hatred of the "bad guys", and boom, page views shoot up as people show up to see the "bad guys" lose, or read about their dastardly deeds and get hacked off. (Reminds me of the political blowhards, which have been using this model for ages. ex: Rush, Bill O'Reilly, etc.)

To be clear, I'm not defending Rightshaven or these guys in any way, but this kind of coverage begins to get cheap and pulpy. (Really, a press release full of BS? Stop the presses!)

Personally I love it, I'm sick of every news outlet covering a story for 15 minutes then dropping it when an interesting video of a cat comes along. Stories don't end that quickly, and Ars gets to the point that we get articles like this, decent analysis of new information from experience, rather than rehashing someone else's press release or AP wire.

I think the follow-up is more than warranted in this case. Righthaven coverage is important because tech vs. copyright is a huge part of Ars' coverage in general. There's no need to cast Ars as some kind of yellow journal rag just because they keep reporting on new developments in important cases.

As for the actual article, between this and the Rightshaven coverage, it's beginning to seem like Ars is creating "bad guys" and running every bit of bad news about them as possible. The readers join in the hatred of the "bad guys", and boom, page views shoot up as people show up to see the "bad guys" lose, or read about their dastardly deeds and get hacked off. (Reminds me of the political blowhards, which have been using this model for ages. ex: Rush, Bill O'Reilly, etc.)

To be clear, I'm not defending Rightshaven or these guys in any way, but this kind of coverage begins to get cheap and pulpy. (Really, a press release full of BS? Stop the presses!)

Except this is totally deserved. Their "open letter" is proof enough that they just won't learn, and are in fact, "bad guys".

Deserved, yes. Justified and Interesting, less so. I mean, I love stories about how baddies are baddies, and stoopid too... I just don't think off Ars Technica as present in that space: that mainly entertainment, not information, and not education.

If we unpack some of the "crap" we get..."We didn't do anything bad. It's all Aaron's fault. Our customers are okay (we hope). We got great security now (we hope). Please buy our $hit. And please! Please! Make the butthurt go away."

The thing that bothers me the most is the rootkit stuff. It's basically them saying "yeah, we're producing cutting-edge malware, for the purpose of government consumption. And we're very proud of that".

Same crap as with crypto. Big brother rules, and there's nothing we can do about it...

While I've enjoyed this series so far, this line right here kinda kills it for me:

PeterB wrote:

Rather, they concerns were that the company was developing these undetectable rootkits, selling (or at least, attempting to sell) them to defense contractors, at which point they could be sold to essentially any agency for any purpose—they could just as well be used to spy on domestic dissidents as they could on foreign powers.

If you seriously feel the need to lodge this form of complaint, an article about HBGary isn't the place to do it. You're worried about government transparency. It's a fair concern, but needs to be addressed as a seperate point more directed at the government than at HBGary. This was the part of the article I lost focus, but only because I think you lost focus when you shifted the eye off HBGary and onto the gov't.

I do agree that anyone should concern themselves with this, but how exactly do you maintain transparency with agencies that, by nature, do possess a valid need for secrecy?

Except this is totally deserved. Their "open letter" is proof enough that they just won't learn, and are in fact, "bad guys".

My point: you missed it.

His point: you don't have one.

It's not "cheap and pulpy" to write a follow-up about a major story when the company involved officially responds to the story (notice how the company waited quite a while, hoping to re-frame the debate virtually unnoticed?) No "presses" were "stopped" (to use *your* cheap and pulpy overblown metaphor).

If only we lived in a world where more news outlets actually practised journalism, and pointed out where (exactly where, with references) press releases were lying. But no, we live in the world of churnalism, where press releases are routinely simply recycled. *That* is "cheap and pulpy".

'were now back in operation with "even stronger cyber defense mechanisms."'

Changed the password did you?

I would laugh if Anon cracked their "defence mechanism" again.

You may get to laugh, the whole open letter feels more like a "yeah, that was a fluke, we've fixed things and we're not scared of Anonymous now, so come and get us if you can little boys!" I totally won't be surprised if we hear about Anonymous hacking HBGary _again_ within a week.

If you seriously feel the need to lodge this form of complaint, an article about HBGary isn't the place to do it. You're worried about government transparency. It's a fair concern, but needs to be addressed as a seperate point more directed at the government than at HBGary. This was the part of the article I lost focus, but only because I think you lost focus when you shifted the eye off HBGary and onto the gov't.I do agree that anyone should concern themselves with this, but how exactly do you maintain transparency with agencies that, by nature, do possess a valid need for secrecy?

If you seriously feel the need to lodge this form of complaint, a comment from a nobody sans credentials under a well-written follow-up to an overall excellently covered debacle from the top source on said story isn't the place to do it. You're worried about who knows what. While no one cares at all, this should be addressed as separate comment to yourself while sitting on the crapper, unless it causes you to lose focus, in which case, the shit you're taking is far more interesting. I agree that no one should concern themselves with this, but how exactly do you maintain such regularity when fecal matter, as evidenced by your own post, possesses the ability to exit both orifices at once?

I agree with previous posters that Anon hitting them again would be hilarious. But this time it should be lots easier; its a sure bet that Anon put more than one backdoor back into the system when they were there.

While I've enjoyed this series so far, this line right here kinda kills it for me:

PeterB wrote:

Rather, they concerns were that the company was developing these undetectable rootkits, selling (or at least, attempting to sell) them to defense contractors, at which point they could be sold to essentially any agency for any purpose—they could just as well be used to spy on domestic dissidents as they could on foreign powers.

If you seriously feel the need to lodge this form of complaint, an article about HBGary isn't the place to do it. You're worried about government transparency. It's a fair concern, but needs to be addressed as a seperate point more directed at the government than at HBGary. This was the part of the article I lost focus, but only because I think you lost focus when you shifted the eye off HBGary and onto the gov't.

I do agree that anyone should concern themselves with this, but how exactly do you maintain transparency with agencies that, by nature, do possess a valid need for secrecy?

To go into hyperanalytical mode here, the five points laid out in HBGary's release were covered in the article. Two were irrefutable claims, with facts to back them (as pointed out in the article). The other three points were of contentious nature (as also pointed out in the article). The author then went to detail only these three points and illustrated just why those three points were contentious, with as much proof to back it as possible (e.g. screen captures of leaked emails).

Of the three points, the final one (the development of the rootkit and the reason behind it), lacked the most solid evidence to be provided because, allegedly, the rootkit was being sold to the government and we all know the government doesn't release such trivial information as that. So, without any solid facts to back up that contentious point, speculation was all that was left. The speculation he provided was short--only one paragraph in length. The speculation was also limited in scope and was narrowly tailored to the fact that it could be used against anyone instead of just "The Bad Guys".

So given the brevity of that speculative passage and the overall coverage of the article with factual backing, to "lose focus" because of that one short bit--which was far from tinfoil hat territory in it's scope--is actually kind of petty, in my views.

Almost like you were looking specifically for a reason to criticize the author.

Good article. Really enjoyed all the HBGary coverage as did a co-worker.

As for this being some sort of dogpile on HBGary, that's called "journalism". Real journalists used to do more than just interview talking heads and recycle press releases. The HBGary coverage shows actual analysis of primary sources (those released emails). Following up on a story like that is acting responsibly. You've reported on an event and now you're reporting on how that event affected it's participants. In this case, HBGary doesn't seem to have learned anything at all and they even seem to be trying to kick the hornets nest again.

When you catch a group doing something underhanded, report it, and then ignore it the next time the group does something underhanded, that would be irresponsible reporting.

Except this is totally deserved. Their "open letter" is proof enough that they just won't learn, and are in fact, "bad guys".

My point: you missed it.

His point: you don't have one.

It's not "cheap and pulpy" to write a follow-up about a major story when the company involved officially responds to the story (notice how the company waited quite a while, hoping to re-frame the debate virtually unnoticed?) No "presses" were "stopped" (to use *your* cheap and pulpy overblown metaphor).

If only we lived in a world where more news outlets actually practised journalism, and pointed out where (exactly where, with references) press releases were lying. But no, we live in the world of churnalism, where press releases are routinely simply recycled. *That* is "cheap and pulpy".

Exactly. It would have been poor journalism to not report on a company's official response to a series of previously published articles.

Well done Ars! This is some of the best journalism I've seen anywhere in a long time. The world would be a better place if all stories were this thoroughly investigated. Make no mistake about it, this is a major story. It will go down in history as the first major cyber-war between a corporation and a loose collective of individuals.

As for the contents of that open letter, wow! They should just put down the shovel now instead of continuing to dig themselves deeper. I wouldn't be surprised if this eventually brings down HBGary Federal entirely.

Except this is totally deserved. Their "open letter" is proof enough that they just won't learn, and are in fact, "bad guys".

My point: you missed it.

His point: you don't have one.

It's not "cheap and pulpy" to write a follow-up about a major story when the company involved officially responds to the story (notice how the company waited quite a while, hoping to re-frame the debate virtually unnoticed?) No "presses" were "stopped" (to use *your* cheap and pulpy overblown metaphor).

If only we lived in a world where more news outlets actually practised journalism, and pointed out where (exactly where, with references) press releases were lying. But no, we live in the world of churnalism, where press releases are routinely simply recycled. *That* is "cheap and pulpy".

Exactly. It would have been poor journalism to not report on a company's official response to a series of previously published articles.

Well done Ars! This is some of the best journalism I've seen anywhere in a long time. The world would be a better place if all stories were this thoroughly investigated. Make no mistake about it, this is a major story. It will go down in history as the first major cyber-war between a corporation and a loose collective of individuals.

As for the contents of that open letter, wow! They should just put down the shovel now instead of continuing to dig themselves deeper. I wouldn't be surprised if this eventually brings down HBGary Federal entirely.

Thank you. It is a major story, and we're not going to let people nag us out of coverage. Cheers!