Share this story

As she was being confirmed as Secretary of State, Hillary Clinton contacted Colin Powell to ask him about his use of a Blackberry while in the same role. According to a Federal Bureau of Investigations memorandum published today (PDF), Powell warned Clinton that if it became public that she was using a Blackberry to "do business," her e-mails would be treated as "official" record and be subject to the law.

"Be very careful," Powell said according to the FBI. "I got around it all by not saying much and not using systems that captured the data."

Clinton told the FBI that she didn't factor Powell's advice into her decision to use a personal mail server—a statement that seems obvious based on the tens of thousands of e-mails now being published as the result of lawsuits, congressional and FBI investigations, and Freedom of Information Act requests. Just how far she deviated from that advice is evident in the detailed history gathered by the FBI. Their information on the Clintons' e-mail infrastructure dates back to Hillary Clinton's tenure in the US Senate, and this new release shows how that infrastructure was intertwined with the information technology used by former president Bill Clinton's staff.

Perhaps Clinton's troubles began when she switched from a Blackberry-hosted e-mail account to an account on her Clintonemail.com domain—a domain hosted on an Apple Power Mac "G4 or G5" tower running in the Clintons' Chappaqua, New York residence. The switch to the Power Mac as a server occurred the same month she exchanged messages with Powell.

Step 1: Power Mac

The Power Mac, originally purchased in 2007 by former President Clinton's aide Justin Cooper, had acted as the server for presidentclinton.com and wjcoffice.com. Cooper managed most of the technology support for Bill Clinton and took charge of setting up Hillary Clinton's new personal mail system on the Power Mac, which sat alongside a firewall and network switching hardware in the basement of the Clintons' home. Accounts were set up for Secretary Clinton and her staff by her husband's staff.

But the Power Mac was having difficulty handling the additional load created by Blackberry usage from Secretary Clinton and her staff, so a decision was made quickly to upgrade the server hardware. Secretary Clinton's deputy chief of staff at the State Department, Huma Abedin, connected Cooper with Brian Pagliano, who had worked in IT for the secretary's 2008 presidential campaign. Cooper inquired with Pagliano about getting some of the campaign's computer hardware as a replacement for the Power Mac, and Pagliano was in the process of selling the equipment off.

Step 2: Dude, you’re getting two Dells

Enlarge/ A Dell PowerEdge 2900, the Clintons' Exchange server for the majority of Hillary Clinton's tenure as Secretary of State.

It was kismet, and in March of 2009, Pagliano delivered two servers to Chappaqua—a Dell PowerEdge 2900 running Windows Server and Microsoft Exchange and a Dell PowerEdge 1950 running Blackberry Enterprise Server (BES). Cooper and Pagliano together acquired additional network and storage hardware. Initially, Pagliano said, he believed the servers were for President Clinton and not for the Secretary.

Pagliano acquired an SSL certificate for the mail server to provide added security for remote e-mail access at that time, and the whole configuration was set up in the Clintons' basement. The Power Mac was converted into a workstation for use by the Clinton household staff, and its contents were eventually backed up to an iMac.

Hillary Clinton said that she was unaware that any of this was going on and that she was only vaguely aware that there was now server hardware in the basement.

Backups of the e-mail server were stored to an external Seagate hard drive. Pagliano told the FBI he did differential backups once a day and a full backup weekly. By June of 2011, the backups were getting to be too much for the external drive, and Pagliano upgraded storage to a Cisco network-attached storage (NAS) system.

Sometime in 2013, Pagliano (who would later get immunity from prosecution) started looking to find a new job. That, and "user limitations and reliability concerns" about the server, led staff for both Secretary and President Clinton to start looking to outsource the whole e-mail thing. According to Secretary Clinton, the move to a hosted service was initiated by President Clinton's staff.

Step 3: A hosted Dell private server

Platte River Networks was hired to set up the new hosted mail server, which would run in an Equinix data center in Secaucus, New Jersey. In June of 2013, a PRN employee came and retrieved the server hardware in Chappaqua, taking it to the data center to migrate the software and contents to virtual machines running on a Dell PowerEdge R620. A Datto SIRIS 2000 backup device was set up in the rack with the server, along with a CloudJacket intrusion detection system, two Dell network switches, and two Fortinet Fortigate 80C firewalls. The server ran e-mail for multiple Clinton domains, including Secretary Clinton's clintonemail.com accounts. The Dell server configured by Pagliano remained in the server cage and wasn't fully decommissioned until December 2013.

While this configuration was undoubtedly more secure than a Power Mac in the Clintons' basement, there were a few hiccups. First, the Clintons had requested, according to a PRN employee interviewed by the FBI, that the contents of the server be encrypted so that only mail recipients could read the content. This was not done, largely so that PRN technicians could "troubleshoot problems occurring within user accounts," the FBI memo reports. Also, while the Clintons had requested only local backups, the Datto appliance initially also used Datto's secure cloud backup service until August of 2015.

Share this story

Sean Gallagher
Sean is Ars Technica's IT and National Security Editor. A former Navy officer, systems administrator, and network systems integrator with 20 years of IT journalism experience, he lives and works in Baltimore, Maryland. Emailsean.gallagher@arstechnica.com//Twitter@thepacketrat

None of this means what she did was ok, but it's also hard to not look askance at the relentless witchhunting when it's placed in that broader context.

You must have missed the e-mails where George Soros was directing Clinton to perform certain duties in certain countries to quell uprisings. You are looking at this whole thing from the wrong angle - look at the bigger picture that these e-mails have uncovered. A civilian giving orders to a Secretary of State of the US government... You don't see a problem with that?

None of this means what she did was ok, but it's also hard to not look askance at the relentless witchhunting when it's placed in that broader context.

You must have missed the e-mails where George Soros was directing Clinton to perform certain duties in certain countries to quell uprisings. You are looking at this whole thing from the wrong angle - look at the bigger picture that these e-mails have uncovered. A civilian giving orders to a Secretary of State of the US government... You don't see a problem with that?

I'm sure he would if the party affiliation of the individual in question were a Republican.

You cannot objectively look at the sequence of events here and think she isn't lying through her teeth in an attempt to hide something. Whether what she is hiding was really that big of a deal is irrelevant...it simply highlights the reason that nearly 70% of people in the country don't trust her.

I can objectively do it, but to you that would be of little matter and an exercise that I doubt you have really tried yourself. You don't like the Clintons, I get it, we all get it.

Nevertheless, Clinton is the only thing standing in the way of a Trump presidency. Now, you want to talk about lies? Is Trump more attractive because his lies are easily seen as just every day rantings? I'm not sure that makes him any more fit for the job than Hillary.

The question isn't whether State's email system was compromised. It was.

The question is whether there was any intention to skirt the legal requirements for security and confidentiality. I don't believe Hillary had the technical savvy to even begin to think about that.

Also, despite Comey's caustic remarks to Congress about recklessness, etc., let's remember that he's not exactly credible, either, when it comes to technology. I mean, he's the same guy who thinks the government should have a backdoor into what would otherwise be secure private systems.

Given the reason Ars used a Power Macintosh G4 picture for the article, have we all forgotten how Apple marketed that machine? Watch the commercial below (and watch out for toxic levels of irony and marketing hype). Here's the script for the commercial:

Quote:

“For the first time in history, a personal computer has been classified as a weapon by the US government. With the power to perform over 1 billion calculations per second, the Pentagon wants to ensure that the new Power Macintosh G4 does not fall into the wrong hands…”

(Apple's crazy claim was based on a federal regulation that a computer over a certain level of power was subject to weapons export controls. It said much more about how fast regulations go out of date, then how powerful a Mac was.)

I remembered the opposite. Many, especially audio professionals, were incensed at the amount of fan noise the G4 towers made. If you wanted to use a G4 for professional audio in a recording studio you had to stick it in the next room and run the cables through the wall into the actual studio, so that the G4's incessant fan noise wouldn't ruin a recording.

I just wanted to make you aware that after about 200 posts to Apple's own "G4 Usage" forum at Apple.com from owners fed up with the noise and wanting official remedies from Apple, a group of Owners formed at email-list and now a website to organize Owners from all over the world to share info on what Apple is saying, and what resolutions people have been getting. We are tracking Owner's Apple Case #s, and we are writing a letter to the Senior VP of AppleCare, along with Steve Jobs. We are also mounting a publicity campaign via videos -- not parodies of the Switch campaign, but videos that explain the frustrations of professionals in music production, video, medical research, etc, that find the G4 just too noisy to work with.

The best thing about the G4 tower? The outrage over the noise level forced Apple to completely redesign the case, and what we got next was the classic Power Mac G5/Mac Pro tower design, with the cooling system redesigned for extreme quiet, which worked so well that Apple shipped that design for almost 10 and a half years. I still have an old Mac Pro under my desk because it has been so reliable and quiet.

None of this means what she did was ok, but it's also hard to not look askance at the relentless witchhunting when it's placed in that broader context.

You must have missed the e-mails where George Soros was directing Clinton to perform certain duties in certain countries to quell uprisings. You are looking at this whole thing from the wrong angle - look at the bigger picture that these e-mails have uncovered. A civilian giving orders to a Secretary of State of the US government... You don't see a problem with that?

You talking about the emails where he, as an American Citizen, requests that attention be paid by the diplomatic office of his government to actions or activities that it may not have been aware of, or which it had not paid sufficient enough attention to? You know, like a whole bunch of us did to the head of the FCC a while back? Or we all did protesting the TPP?

The difference, the ONLY difference, is that Soros is known by Clinton, so he's established he's not going to waste her time with conspiracy theories. Kinda like how I'm "known" by the TPB crowd, so I don't have to do a captcha when I upload there, or how I have a whole bunch of extra features available to me on youtube because I've a longstanding account that's stayed in excellent status with a good solid number of views. Basically I'm a known stable quality. I'm not quite at pewtepie levels yet, but you know that someone like him has good contacts at Youtube. Same thing.

...It is NSA's responsibility to provide communications for the heads of departments, including the Secretary of State. Clinton supposedly asked for a secure Blackberry like Obama's, but the NSA refused, siting cost. The NSA seems to think the Secretary of State only needs the security found within the SCIF in the State Department offices, and not portable security. Really? No one travels more than the Secretary of State...

That's the headline I keep reading. And it looks like you've read it too. What they don't tell us is that instead they wanted her to use a General Dynamics Sectéra Edge. Which while NSA approved for mobile SCIF classified communication, it wasn't cool enough for Hillary.

None of this means what she did was ok, but it's also hard to not look askance at the relentless witchhunting when it's placed in that broader context.

You must have missed the e-mails where George Soros was directing Clinton to perform certain duties in certain countries to quell uprisings. You are looking at this whole thing from the wrong angle - look at the bigger picture that these e-mails have uncovered. A civilian giving orders to a Secretary of State of the US government... You don't see a problem with that?

also question.If Soros can just boss her around.......and she's so crooked and nefarious that she can control the FBI, Republican congressional investigations created solely to attack her, and is able to seamlessly manage a full international conspiracy spanning hundreds of thousands of people in multiple countries, for 20+ years, despite many being equally, or more 'powerful' and 'connected' as her, assassinating at will ........ while also being apparently utterly incompetent at everything but lucking through (like a sober Mallory Archer)...

... How fucking scary must Soros be to be able to order HER around! I know! Maybe he's the one that made her sick and unless she does his bidding she doesn't get the antidote!!!!!!!

...It is NSA's responsibility to provide communications for the heads of departments, including the Secretary of State. Clinton supposedly asked for a secure Blackberry like Obama's, but the NSA refused, siting cost. The NSA seems to think the Secretary of State only needs the security found within the SCIF in the State Department offices, and not portable security. Really? No one travels more than the Secretary of State...

That's the headline I keep reading. And it looks like you've read it too. What they don't tell us is that instead they wanted her to use this smartphone. While NSA approved for classified mobile communications, It wasn't cool enough for Hillary.

...It is NSA's responsibility to provide communications for the heads of departments, including the Secretary of State. Clinton supposedly asked for a secure Blackberry like Obama's, but the NSA refused, siting cost. The NSA seems to think the Secretary of State only needs the security found within the SCIF in the State Department offices, and not portable security. Really? No one travels more than the Secretary of State...

That's the headline I keep reading. And it looks like you've read it too. What they don't tell us is that instead they wanted her to use this smartphone. While NSA approved for classified mobile communications, It wasn't cool enough for Hillary.

...It is NSA's responsibility to provide communications for the heads of departments, including the Secretary of State. Clinton supposedly asked for a secure Blackberry like Obama's, but the NSA refused, siting cost. The NSA seems to think the Secretary of State only needs the security found within the SCIF in the State Department offices, and not portable security. Really? No one travels more than the Secretary of State...

That's the headline I keep reading. And it looks like you've read it too. What they don't tell us is that instead they wanted her to use this smartphone. While NSA approved for classified mobile communications, It wasn't cool enough for Hillary.

I think it was less a case of its lack of 'coolness factor' and more the heavy dose of the 'overall shitiness factor'.

So fuck it, i'm just going to use my Blackberry!"

That's the response i'd expect from Lindsay Lohan (followed by "Don't they know who i am?! I'm a celebrity!"" not Secratery of State Hillary Clinton.

No, the fact it's essentially not fit for purpose is why. That and the predecessor in her role used one, so the tech was physically capable of it, it was just a denial based on essentially whim (or perhaps political calculation) as to why one of the leading members of the government (4th in succession, 10th in the overall order of precedence

Even if she chose to use her many Blackberry's and iPad to access her unclassified mail... she sure as hell should have been carrying the NSA approved SCIF mobile device too, for when she needed to conduct mobile classified communications. The SCIF phone has totally separate sides to it for both regular as well as classified communications.

I mean what the Hell... I carry two phones, 1 my personal smartphone for mobile internet and keeping in touch with close friends and family. And the other a lowly feature phone that's my public phone number that I give out to everyone else and call them from.

It's not that hard. She didn't need to used the un-secured side of the SKIF phone at all. She could still use her crackberry for that. But for secure and classified communications, that's what the SKIF phone was for. But it's old! isn't an excuse not to use it. Even Kerry's latest hardened phone is only a Galaxy S4. You use what you've got available to you.

...It is NSA's responsibility to provide communications for the heads of departments, including the Secretary of State. Clinton supposedly asked for a secure Blackberry like Obama's, but the NSA refused, siting cost. The NSA seems to think the Secretary of State only needs the security found within the SCIF in the State Department offices, and not portable security. Really? No one travels more than the Secretary of State...

That's the headline I keep reading. And it looks like you've read it too. What they don't tell us is that instead they wanted her to use this smartphone. While NSA approved for classified mobile communications, It wasn't cool enough for Hillary.

I think it was less a case of its lack of 'coolness factor' and more the heavy dose of the 'overall shitiness factor'.

So fuck it, i'm just going to use my Blackberry!"

That's the response i'd expect from Lindsay Lohan (followed by "Don't they know who i am?! I'm a celebrity!"" not Secratery of State Hillary Clinton.

No, the fact it's essentially not fit for purpose is why. That and the predecessor in her role used one, so the tech was physically capable of it, it was just a denial based on essentially whim (or perhaps political calculation) as to why one of the leading members of the government (4th in succession, 10th in the overall order of precedence

"The latest batch of State Department documents recently obtained by Judicial Watch in a Freedom of Information Act (FOIA) lawsuit regarding Hillary Clinton’s use of a private email server while serving as Secretary of State revealed that she repeatedly sought to obtain an uncrackable “Blackberry-like communications” device to be used by her and her staff in their Sensitive Compartmented Information Facility offices. But the Secretary of State was rebuffed by the National Security Agency due to security and cost concerns.

The FOIA-obtained documents detail that Donald R. Reid as the State Department’s Senior Coordinator for Security Infrastructure at the Bureau of Diplomatic Security knew the Blackberry (BB) that President Obama carried wasn’t designed to be secure enough to send classified email — only to protect its microphone from being remotely hijacked by cyber-spies, according to Wired magazine.

But Reid persisted in a February 18, 2009 email to the NSA, because Secretary Clinton:

does not use a personal computer so our view of someone wedded to their email (why doesn’t she use her desktop when in SCIF?) doesn’t fit this scenario … during the campaign she was urged to keep in contact with thousands via a BB … once she got the hang of it she was hooked … now everyday [sic], she feels hamstrung because she has to lock her BB up … she does go out several times a day to an office they have crafted for her outside the SCIF and plays email catch up … Cheryl Mills and others who are dedicated BB addicts are frustrated because they too are not near their desktop very often during the working day.

The Secretary of State and her staff had developed a “personal comfort” using Blackberry technology during Clinton’s failed 2008 presidential campaign, and were angry because Blackberry security waivers were issued to Secretary of State Condoleezza Rice and her staff. But the NSA stated that “use expanded to an unmanageable number of users from a security perspective, so those waivers were phased out and Blackberry use was not allowed in her suite.”

For more special secure mobile hardware, the NSA suggested Secretary Clinton use a “Sectéra Edge” phone developed by General Dynamics under the direction of the NSA as part of the Secure Mobile Environment Portable Electronic Device (SME PED) program. The Sectéra Edge cost $4,750 per unit and ran a special hardened version of Windows CE software that supported the necessary protocols to access classified information on top secret government networks and place encrypted calls to other similarly trusted devices, according to by Ziff Davis Media’s Extreme Tech."

So yes, basically exactly what I said, albeit framed through an extremely partisan blog, written by a businessman who doesn't understand tech.Odds that the denial of the waiver came from someone that would have benefited from the in-house developed shitty phone you're championing that was really unsuitable for the job it was supposed to do? Pretty good probably.

Psst instead of quoting an investment banker writing for a site that's been so in the pocket for her opponent that its head now runs his campaign, look down a little, that story, written by Ars, is the first 'related story'. It's far better written and less full of assumptions and loaded phrasings. Also explains why no-one wanted to use that phone, which (iirc) is backed up in the comments by people who have experience with it.

It's a breach of protocol. She mishandled classified information she otherwise had clearance to see. It's about equivalent to discussing state secrets over an unsecured phone line in a seedy motel, or leaving top secret information lying out on your kitchen table while you have your friends over for a BBQ. It was incredibly stupid of her, and she's lucky there's only theoretical evidence of a possibility of a leak, but it's not criminal.

I agree with Comey's conclusion on the matter. It's something any "regular" person would've been fired over, probably blackballed from any sensitive government position for life, though it's nothing anyone would go to jail over.

A missing piece of this whole conversation is what IT would be in place for the Secretary of State instead of personal email servers. Government servers that have been known to be all too easily hacked? And, just which department has the responsibility for government security? As with all bureaucracies, the responsibility is spread among many departments, including the FBI.

It is NSA's responsibility to provide communications for the heads of departments, including the Secretary of State. Clinton supposedly asked for a secure Blackberry like Obama's, but the NSA refused, siting cost. The NSA seems to think the Secretary of State only needs the security found within the SCIF in the State Department offices, and not portable security. Really? No one travels more than the Secretary of State.

John Kerry's mobile systems (now that they finally have them) were updated just weeks ago, and if you look at what he now has, you will find that those systems are five years behind the times.

I am much more concerned about IT security within all departments of the federal government than I am what Clinton did or did not do.

I'm concerned about IT security, which makes me very concerned about finally funding IT so it can succeed. Every government organization I've worked with, even with top level universities, fund their landscaping better than their IT. And that means the buck stops with whatever boss determines funding.

Please don't tell me this is about the taxpayer deciding funding for IT, because we know that Social Security was better prepared for Y2K than almost any other government department. If the unknown director of Social Security could wrangle a decent IT budget (past tense on that), then it can still be done by much bigger names & departments. (Not singling out one department, too many hacks to choose from)

None of this means what she did was ok, but it's also hard to not look askance at the relentless witchhunting when it's placed in that broader context.

...

My personal evolution on this issue has gone from "having a privately controlled email server sounds really really bad, and was probably done to avoid monitoring! I'm really upset about this!" to "wow, these allegations sound extremely serious!" to "oh, those allegations were not really true at all" to "yikes, this again? how much more whining and knashing of the the teeth am I going to have to put up with?" If this had been any other politican, like, literally any other politician would we have heard more than a week or two about it? Would we have the FBI releasing their investigation documents to the public? Would all of Clinton's emails been open to the public like this? The amount of transparency, the lack of smoking guns, and the irrationally emotional anger have made me completely turn around on this issue.

The reason it keeps coming back is that each new revelation seems to reveal more lies and more proof of lies by Hillary Clinton. You suggest if it was any other politician it would be instantly forgotten. Not exactly. Not if they stood a very good chance of being the next president of the United States. And certainly not if they had the same background of corruption, lying, and disastrous job performance as Clinton does (getting Americans killed in Benghazi and then lying to their families about it, her lies about being under sniper attack on the tarmac in the Balkans years ago, etc etc). Nixon was forced to resign for far less dishonesty than this woman has been caught in. So yes, it is a big deal, and it should be. Not only did she take the classified workflow outside of the secure state department infrastructure, she did it to avoid accountability and just exactly the kind of scandal that would ensue if it was ever found out, which it obviously was. She put national security at risk for her own political gain, and then lied about it repeatedly on many occasions and in all kinds of settings. Not only did she commit crimes and SHOULD have been charged by DOJ (her hubby's little illicit chit-chat w/ Lynch on the Phoenix tarmac notwithstanding), but she demonstrated by all she has done she doesn't have the one thing a real president needs: good judgement. Plenty of other things as well, honesty, etc, should also be requirements, but generally aren't, lately. But having better judgement than a 2 year old is crucial, and she's proven she hasn't got that.

A recap ( Comey's testimony) of just some of the lies told by Clinton, to both the public, Congress, and the FBI, about her emails, server, etc :

...getting Americans killed in Benghazi and then lying to their families about it

I can understand the arguments and disagreements surrounding the email issue, but I still can't believe there are people who view Benghazi as anything other than a crass witch hunt/character assassination attempt against an opposing candidate for the presidency.

How many investigations did the Republicans field for it? Around 9 or so, wasn't it, including 2 depositions? And what did they gain for all that time and money spent? Absolutely no evidence of any wrongdoing on Hillary's part. Of course, that didn't stop them from putting one of the victim's mother up on stage, so she could scream "HILLARY CLINTON MURDERED MY SON" in front of the millions of people watching the RNC. Can't let a good manufactured controversy go to waste, after all.

The Benghazi debacle was nothing more than a crass, cynical stunt to harm reputations. By the end of it, I was surprised they hadn't dug up the corpses of the fallen, put 'em on puppet strings, and had them perform a little song and dance number about Hillary on the National Mall.

The reason it keeps coming back is that each new revelation seems to reveal more lies and more proof of lies by Hillary Clinton. You suggest if it was any other politician it would be instantly forgotten. Not exactly. Not if they stood a very good chance of being the next president of the United States. And certainly not if they had the same background of corruption, lying, and disastrous job performance as Clinton does (getting Americans killed in Benghazi and then lying to their families about it, her lies about being under sniper attack on the tarmac in the Balkans years ago, etc etc). Nixon was forced to resign for far less dishonesty than this woman has been caught in. So yes, it is a big deal, and it should be. Not only did she take the classified workflow outside of the secure state department infrastructure, she did it to avoid accountability and just exactly the kind of scandal that would ensue if it was ever found out, which it obviously was. She put national security at risk for her own political gain, and then lied about it repeatedly on many occasions and in all kinds of settings. Not only did she commit crimes and SHOULD have been charged by DOJ (her hubby's little illicit chit-chat w/ Lynch on the Phoenix tarmac notwithstanding), but she demonstrated by all she has done she doesn't have the one thing a real president needs: good judgement. Plenty of other things as well, honesty, etc, should also be requirements, but generally aren't, lately. But having better judgement than a 2 year old is crucial, and she's proven she hasn't got that.

Could you at least break your thoughts into paragraphs? Also, back up your whining with actual facts. Yeah, that would be nice.

...It is NSA's responsibility to provide communications for the heads of departments, including the Secretary of State. Clinton supposedly asked for a secure Blackberry like Obama's, but the NSA refused, siting cost. The NSA seems to think the Secretary of State only needs the security found within the SCIF in the State Department offices, and not portable security. Really? No one travels more than the Secretary of State...

That's the headline I keep reading. And it looks like you've read it too. What they don't tell us is that instead they wanted her to use this smartphone. While NSA approved for classified mobile communications, It wasn't cool enough for Hillary.

I think it was less a case of its lack of 'coolness factor' and more the heavy dose of the 'overall shitiness factor'.

So fuck it, i'm just going to use my Blackberry!"

That's the response i'd expect from Lindsay Lohan (followed by "Don't they know who i am?! I'm a celebrity!"" not Secratery of State Hillary Clinton.

No, the fact it's essentially not fit for purpose is why. That and the predecessor in her role used one, so the tech was physically capable of it, it was just a denial based on essentially whim (or perhaps political calculation) as to why one of the leading members of the government (4th in succession, 10th in the overall order of precedence

I can't find it but I recall specifically that the number of blackberry users with or without access to classified information in Washington alone was somewhere around 100,000 at one time. That included of course the Department of Defense.

Wow. Not only is this article misleading and poorly composed, it is factually incorrect (pic being one example). At the time this happened was it uncommon for a company to manage their own email servers/hardware? What were BlackBerry recommendations on hosting? Who actually ordered the hardware? Who is PRN and what other clients do they represent?

Wow. Not only is this article misleading and poorly composed, it is factually incorrect (pic being one example). At the time this happened was it uncommon for a company to manage their own email servers/hardware? What were BlackBerry recommendations on hosting? Who actually ordered the hardware? Who is PRN and what other clients do they represent?

In a country where a standing governer running as VP could be found explicitly and intentionally using Yahoo email for the express purpose of avoiding FOIA on relevant government business, and there be no investigation whatsoever… well. Let's just say there's an exceedingly strong whiff of double standards in the air.

I'm not fond of this private server crap. I think it's bullshit and it never should have been allowed in the first place. She should have simply been told that it's not permissible, whatsoever. But I also think the classified email issues are red herrings in the context of the use of private servers, as they would have been just as much an issue on State Department non classified servers.

And I think that it's been made abundantly clear that the tools to do business over email and modern mobile computing were extremely lacking, outside of a solution like this, and what tools were available were purposefully withheld over what sounds like ridiculous political fighting under the guise of bureaucracy.

None of this means what she did was ok, but it's also hard to not look askance at the relentless witchhunting when it's placed in that broader context.

Personally I've reached a point where I'm done caring on the topic. There doesn't seem to be any kind of smoking gun, just a lot of hemming and hawing. Normally I would care about this, but honestly I'm a bit inured at this point. Where is the show of her using these specifically to avoid FOIA on work material actually relevant to FOIA?

That's really the only true relevant question when it comes to moving to private servers. Classified material isn't supposed to be on unclassified government servers either, so the attempt to focus on that (mostly with retroactive or improperly labeled material and a few other issues) really seems awkward when we're supposed to care about the private servers as if they're damning.

Most interesting to me was confirmation that the server was breached. Unknown parties accessed it from TOR multiple times.

From your link, an individual email account on the server was breached.

This happens all the time, for varying reasons, mostly due to a phishing compromise of the account, and occasionally due to password re-use and related vectors of compromise. While it's bad for the individual account's contents, it's absolutely irrelevant beyond that.

If that's the worst they can find then personally I'm actually impressed. I was expecting that the server(s) had been root/fully compromised at least once, given how they get perennially described. If that turns out to not be the case, then they've actually been run better and more securely than the State Department's [at least non-classified] servers, from all reports.

Look, getting all up in arms over crap like that link is why people like me are no longer convinced there's anything here worth paying attention to. I'm actually willing to listen if there's some kind of smoking gun, but that's some petty bullshit right there.

The smoking gun is that Clinton had the incredibly bad judgement to use the private server at all. Number 4 in line for the presidency and she does that? There are thousands of others at lesser levels than her all of whom never would have the carelessness to do it and never, never ever, would have walked scot free if they had. I had a security clearance and the NSA does not mess around with them. Losing ALL clearance would have happened in every case, felony charges for nearly anybody but Clinton and so on.

So if you want a smoking gun that is it - horrific judgement and gross negligence followed by political protection of the highest order.

I am much more concerned about IT security within all departments of the federal government than I am what Clinton did or did not do.

This is the point anyone who cares about the country should be making, and I really wish Hillary had raised it early on. Federal IT is bad not because of the usual right-wing tropes about government workers but because there are too many barriers enshrined in federal law and policy. Things like procurement, hiring, and even the simple ability to deploy an application have slow, expensive processes full of counter-productive incentives. The pay-scale for federal staff tops out well below the private sector, there's been a couple decades of Congress trying to encourage outsourcing (I'm sure it's just a coincidence that large contracting companies can make campaign donations), and a lot of senior management and policy have tried to treat IT as a purchase rather than a skill to be developed, all of which means that the federal workforce is aging and the best people are routinely asking themselves whether they believe in their agency's mission enough to keep turning down a hefty pay raise. GitHub's Ben Balter, a former Presidential Innovation Fellow, has written a lot about this – see What’s next for federal IT policy, IMHO, Three things you learn going from the most bureaucratic organization in the world to the least, Want to innovate government? Focus on culture, etc.

This has already been a big deal during the Obama administration and I think it's going to become critical for the next president as both our dependencies on IT continue to increase – remember that due to decades of budget cuts, many agencies are still relatively early in the migration to fully electronic processes – and the demands increase, both for general worker productivity and especially for across-the-board security improvements as the sophistication of attacks has gone up. Security is one of the hardest parts of IT because it's not a commodity which you can purchase, requires broader skills and constant adjustment, and the field is full of hucksters peddling purchases or bureaucratic process as easy solutions. The low federal pay-scale is especially bad since there's so much private sector demand, which means that it's hard to keep skilled practitioners on staff and that reduces the pool of qualified people getting hired into management.

This is the kind of thing people should be asking the candidates to talk about but due to the prolonged bad-faith attempts to trump up scandals from things like these emails it's really hard to see any sort of honest policy discussion breaking out. Every citizen should care about changing that dynamic since in addition to the areas where the failures are themselves major crises everywhere else they're behind the scenes making projects more expensive and less successful across the board.

Wow. Not only is this article misleading and poorly composed, it is factually incorrect (pic being one example). At the time this happened was it uncommon for a company to manage their own email servers/hardware? What were BlackBerry recommendations on hosting? Who actually ordered the hardware? Who is PRN and what other clients do they represent?

During the "growing" age of the Internet but before cloud computing (I'd say early 1990's to mid 2000's) it was very easy/common to run your own servers. All you needed was a constant internet connection and a static IP addr.

Wow. Not only is this article misleading and poorly composed, it is factually incorrect (pic being one example). At the time this happened was it uncommon for a company to manage their own email servers/hardware? What were BlackBerry recommendations on hosting? Who actually ordered the hardware? Who is PRN and what other clients do they represent?

During the "growing" age of the Internet but before cloud computing (I'd say early 1990's to mid 2000's) it was very easy/common to run your own servers. All you needed was a constant internet connection and a static IP addr.

And now with cloud computing all you need is a domain and the ability to add an MX record, and you can have the people who built Exchange server run it and monitor it for you for $4 dollars a month per account plus unlimited aliases!

Hillary Clinton didn't need to use her own Blackberry. The State Deptartment had been using Blackberries since 2006, and diplomats overseas had been using them for just as long.

Hillary Clinton didn't need to use a fancy NSA-approved smartphone to access classified data. Whenever she went overseas, she had a team of IT specialists who was able to provide her with ClassNet access, and they're able to do so without any technical support from a US Embassy.

Wow. Not only is this article misleading and poorly composed, it is factually incorrect (pic being one example). At the time this happened was it uncommon for a company to manage their own email servers/hardware? What were BlackBerry recommendations on hosting? Who actually ordered the hardware? Who is PRN and what other clients do they represent?

During the "growing" age of the Internet but before cloud computing (I'd say early 1990's to mid 2000's) it was very easy/common to run your own servers. All you needed was a constant internet connection and a static IP addr.

This was especially common among non-IT centric businesses in my experience – doctors, lawyers, non-profits, etc. would pay a consultant to set something up and give their front-office staff instructions about changing backup tapes, etc. but they didn't want to have to deal with the complexity and expense of a real data center operation, hiring staff, etc. You probably wanted a business cable/DSL connection anyway, buy a copy of Windows Small business Server or OS X Server depending on your tastes and you have everything “done” for a fixed up-front cost. A lot of consultants made good livings doing the same setup for a bunch of clients which weren't quite big enough to have IT staffing or balked at paying someone above desktop-support level.

The biggest things which killed that market were security and disaster recovery, as maintaining an email server became a full-time job and stories about someone losing everything in a hack / fire / flood / etc. became fairly common, coupled with the availability of high-quality services (Google Apps for Your Domain launched in 2006) at prices which were much less than you could match for things like spam filtering, user interface quality, and performance at a scale of less than hundreds of users. Things like PCI or HIPAA accelerated that process by telling entire fields it was no longer a good area to skimp.

By now it's assumed most small operations will use a cloud provider but it took years to establish that the service quality and pricing would stick. By the time Hillary took office, however, that was still in transition. It doesn't surprise me at all that someone – especially someone mid-career or older – would go back to what was familiar when their boss asked them to get something done in a hurry. It's the same process you can find all over the business world where someone has a “mission critical” Access database, Excel file, PHP app on a shared host, etc. because they were told to get it done ASAP and didn't have time to learn something new, especially if this wasn't a core part of their job. It'll just be a temporary fix until we do things the right way…

Wow. Not only is this article misleading and poorly composed, it is factually incorrect (pic being one example). At the time this happened was it uncommon for a company to manage their own email servers/hardware? What were BlackBerry recommendations on hosting? Who actually ordered the hardware? Who is PRN and what other clients do they represent?

During the "growing" age of the Internet but before cloud computing (I'd say early 1990's to mid 2000's) it was very easy/common to run your own servers. All you needed was a constant internet connection and a static IP addr.

This was especially common among non-IT centric businesses in my experience – doctors, lawyers, non-profits, etc. would pay a consultant to set something up and give their front-office staff instructions about changing backup tapes, etc. but they didn't want to have to deal with the complexity and expense of a real data center operation, hiring staff, etc. You probably wanted a business cable/DSL connection anyway, buy a copy of Windows Small business Server or OS X Server depending on your tastes and you have everything “done” for a fixed up-front cost. A lot of consultants made good livings doing the same setup for a bunch of clients which weren't quite big enough to have IT staffing or balked at paying someone above desktop-support level.

The biggest things which killed that market were security and disaster recovery, as maintaining an email server became a full-time job and stories about someone losing everything in a hack / fire / flood / etc. became fairly common, coupled with the availability of high-quality services (Google Apps for Your Domain launched in 2006) at prices which were much less than you could match for things like spam filtering, user interface quality, and performance at a scale of less than hundreds of users. Things like PCI or HIPAA accelerated that process by telling entire fields it was no longer a good area to skimp.

By now it's assumed most small operations will use a cloud provider but it took years to establish that the service quality and pricing would stick. By the time Hillary took office, however, that was still in transition. It doesn't surprise me at all that someone – especially someone mid-career or older – would go back to what was familiar when their boss asked them to get something done in a hurry. It's the same process you can find all over the business world where someone has a “mission critical” Access database, Excel file, PHP app on a shared host, etc. because they were told to get it done ASAP and didn't have time to learn something new, especially if this wasn't a core part of their job. It'll just be a temporary fix until we do things the right way…

Answers in a can. Just shake and spray. Although with the subscription model of clouds in general, people still are going to take the cheapest route.

Most interesting to me was confirmation that the server was breached. Unknown parties accessed it from TOR multiple times.

"multiple times" is 3 times in this case, and it wasn't the server that was breached, it was 1 person's email.

Even if this person was clinton herself, we already know there was not much damaging information stored on this server. And considering this seems more like someone used a weak password or was phished, this is a vulnerability no matter what email provider you're using.

We're going to get into this in a story I'm currently writing (probably for next week, so it's not a Friday newsdumpster move). But it's worth noting THE ENTIRETY OF THE STATE DEPARTMENT'S UNCLAS EMAIL SYSTEM WAS PWNED FOR OVER A YEAR. I'm sorry, did I type that in all-caps? Also, between Chelsea Manning/ Wikileaks and the repeated hacks of State, the White House, etc between 2009 and 2014, it is highly likely that everything short of the TS/SAP stuff (and even some of that) that Clinton touched was already breached.

This does not excuse Clinton and her staff's—I'm looking at you, Jake Sullivan—for the extreme error of passing Top Secret/ Special Access Program classified data back and forth over Blackberries and a non-governmental e-mail system. I would expect that Sullivan, at a minimum, will have his clearance revoked and he will not be getting a job as a national security adviser if Clinton wins the election. Or at least, I think that's a reasonable expectation.

Mr. Gallagher - Nice article, thanks. The technical details of Clinton’s email saga are interesting. The security breaches are troubling, but these are really tangential to the bigger story. The essence of Clinton’s problem is that she side-stepped the State Department email system due to her obsession with hiding her communications, particularly the dealings with the Clinton Foundation, from FOIA requests and subsequent investigations. Government officials’ communications, with narrow exceptions, are subject to FOIA and document archiving laws; Secretary Clinton apparently considered herself above that. Perhaps did not consciously intend to illegally store or transmit classified documents, but that inevitably happened due to her negligence and disdain for FIOA and document archiving rules. Now we see that dealings by her staff with the Clinton Foundation violated her agreement with the Obama administration and clearly demonstrated ethical lapses and conflicts of interest.