Share this story

As she was being confirmed as Secretary of State, Hillary Clinton contacted Colin Powell to ask him about his use of a Blackberry while in the same role. According to a Federal Bureau of Investigations memorandum published today (PDF), Powell warned Clinton that if it became public that she was using a Blackberry to "do business," her e-mails would be treated as "official" record and be subject to the law.

"Be very careful," Powell said according to the FBI. "I got around it all by not saying much and not using systems that captured the data."

Clinton told the FBI that she didn't factor Powell's advice into her decision to use a personal mail server—a statement that seems obvious based on the tens of thousands of e-mails now being published as the result of lawsuits, congressional and FBI investigations, and Freedom of Information Act requests. Just how far she deviated from that advice is evident in the detailed history gathered by the FBI. Their information on the Clintons' e-mail infrastructure dates back to Hillary Clinton's tenure in the US Senate, and this new release shows how that infrastructure was intertwined with the information technology used by former president Bill Clinton's staff.

Perhaps Clinton's troubles began when she switched from a Blackberry-hosted e-mail account to an account on her Clintonemail.com domain—a domain hosted on an Apple Power Mac "G4 or G5" tower running in the Clintons' Chappaqua, New York residence. The switch to the Power Mac as a server occurred the same month she exchanged messages with Powell.

Step 1: Power Mac

The Power Mac, originally purchased in 2007 by former President Clinton's aide Justin Cooper, had acted as the server for presidentclinton.com and wjcoffice.com. Cooper managed most of the technology support for Bill Clinton and took charge of setting up Hillary Clinton's new personal mail system on the Power Mac, which sat alongside a firewall and network switching hardware in the basement of the Clintons' home. Accounts were set up for Secretary Clinton and her staff by her husband's staff.

But the Power Mac was having difficulty handling the additional load created by Blackberry usage from Secretary Clinton and her staff, so a decision was made quickly to upgrade the server hardware. Secretary Clinton's deputy chief of staff at the State Department, Huma Abedin, connected Cooper with Brian Pagliano, who had worked in IT for the secretary's 2008 presidential campaign. Cooper inquired with Pagliano about getting some of the campaign's computer hardware as a replacement for the Power Mac, and Pagliano was in the process of selling the equipment off.

Step 2: Dude, you’re getting two Dells

Enlarge/ A Dell PowerEdge 2900, the Clintons' Exchange server for the majority of Hillary Clinton's tenure as Secretary of State.

It was kismet, and in March of 2009, Pagliano delivered two servers to Chappaqua—a Dell PowerEdge 2900 running Windows Server and Microsoft Exchange and a Dell PowerEdge 1950 running Blackberry Enterprise Server (BES). Cooper and Pagliano together acquired additional network and storage hardware. Initially, Pagliano said, he believed the servers were for President Clinton and not for the Secretary.

Pagliano acquired an SSL certificate for the mail server to provide added security for remote e-mail access at that time, and the whole configuration was set up in the Clintons' basement. The Power Mac was converted into a workstation for use by the Clinton household staff, and its contents were eventually backed up to an iMac.

Hillary Clinton said that she was unaware that any of this was going on and that she was only vaguely aware that there was now server hardware in the basement.

Backups of the e-mail server were stored to an external Seagate hard drive. Pagliano told the FBI he did differential backups once a day and a full backup weekly. By June of 2011, the backups were getting to be too much for the external drive, and Pagliano upgraded storage to a Cisco network-attached storage (NAS) system.

Sometime in 2013, Pagliano (who would later get immunity from prosecution) started looking to find a new job. That, and "user limitations and reliability concerns" about the server, led staff for both Secretary and President Clinton to start looking to outsource the whole e-mail thing. According to Secretary Clinton, the move to a hosted service was initiated by President Clinton's staff.

Step 3: A hosted Dell private server

Platte River Networks was hired to set up the new hosted mail server, which would run in an Equinix data center in Secaucus, New Jersey. In June of 2013, a PRN employee came and retrieved the server hardware in Chappaqua, taking it to the data center to migrate the software and contents to virtual machines running on a Dell PowerEdge R620. A Datto SIRIS 2000 backup device was set up in the rack with the server, along with a CloudJacket intrusion detection system, two Dell network switches, and two Fortinet Fortigate 80C firewalls. The server ran e-mail for multiple Clinton domains, including Secretary Clinton's clintonemail.com accounts. The Dell server configured by Pagliano remained in the server cage and wasn't fully decommissioned until December 2013.

While this configuration was undoubtedly more secure than a Power Mac in the Clintons' basement, there were a few hiccups. First, the Clintons had requested, according to a PRN employee interviewed by the FBI, that the contents of the server be encrypted so that only mail recipients could read the content. This was not done, largely so that PRN technicians could "troubleshoot problems occurring within user accounts," the FBI memo reports. Also, while the Clintons had requested only local backups, the Datto appliance initially also used Datto's secure cloud backup service until August of 2015.

Share this story

Sean Gallagher
Sean is Ars Technica's IT and National Security Editor. A former Navy officer, systems administrator, and network systems integrator with 20 years of IT journalism experience, he lives and works in Baltimore, Maryland. Emailsean.gallagher@arstechnica.com//Twitter@thepacketrat

175 Reader Comments

First, the Clintons had requested, according to a PRN employee interviewed by the FBI, that the contents of the server be encrypted so that only mail recipients could read the content. This was not done, largely so that PRN technicians could "troubleshoot problems occurring within user accounts," the FBI memo reports. Also, while the Clintons had requested only local backups, the Datto appliance initially also used Datto's secure cloud backup service until August of 2015.

Sounds like some of the problem was the contractor not following the procedures established by the client.

First, the Clintons had requested, according to a PRN employee interviewed by the FBI, that the contents of the server be encrypted so that only mail recipients could read the content. This was not done, largely so that PRN technicians could "troubleshoot problems occurring within user accounts," the FBI memo reports. Also, while the Clintons had requested only local backups, the Datto appliance initially also used Datto's secure cloud backup service until August of 2015.

Sounds like some of the problem was the contractor not following the procedures established by the client.

Convenient that the contractor was given full immunity by the justice department over this matter, so I suspect a lot of blame will be headed his way.

The mac shown in the image was last sold around 2002. Why would they buy that in 2007?

And even if that's what they ran, wouldn't it still easily be able to perform as an email server for just a few people?

The FBI said it was either a G4 or a G5. In either case, both were discontinued by 2007. I suspect that either the FBI is wrong about the server, or about when it was acquired. It may have been used for other purposes before being turned into a mail server in 2007.

Just to clarify, the move to a hosted solution - with requested encryption - was initiated after Clinton's tenure as Secretary of State (January 21, 2009 – February 1, 2013) was completed in February, 2013, and FOIA requests were no longer applicable [for e-mail traffic after her resignation, but not for existing records] as she was no longer a government employee.

Edit: Clarification in [brackets] per comments from @diaphanein (thanks). I wasn't referring to stored data but for active communications in Clinton's post-government use of the server.

Just to clarify, the move to a hosted solution - with requested encryption - was initiated after Clinton's tenure as Secretary of State (January 21, 2009 – February 1, 2013) was completed in February, 2013, and FOIA requests were no longer applicable as she was no longer a government employee.

I think that would depend on the scope of the migration. Did they migrate all of the history over to the hosted solution? i.e. Did they migrate the OS, Exchange and BES servers into PRN's datacenter? Or, did they start from scratch with a clean slate, fresh install and no data migration. If it's the former and not the latter, I'd be pretty damned certain it'd still be subject to FOIA requests.

Most interesting to me was confirmation that the server was breached. Unknown parties accessed it from TOR multiple times.

Not sure why you are being down voted on newly revealed information that seems to confirm that one of the servers email accounts was breached.

If you're down voting him, perhaps an explanation as to why?

Probably because we know DOJ email servers have also been breached. He's implying that her servers were less secure and somehow put information in harms way. History seems to show us that it wasn't at any more risk.

In a country where a standing governer running as VP could be found explicitly and intentionally using Yahoo email for the express purpose of avoiding FOIA on relevant government business, and there be no investigation whatsoever… well. Let's just say there's an exceedingly strong whiff of double standards in the air.

I'm not fond of this private server crap. I think it's bullshit and it never should have been allowed in the first place. She should have simply been told that it's not permissible, whatsoever. But I also think the classified email issues are red herrings in the context of the use of private servers, as they would have been just as much an issue on State Department non classified servers.

And I think that it's been made abundantly clear that the tools to do business over email and modern mobile computing were extremely lacking, outside of a solution like this, and what tools were available were purposefully withheld over what sounds like ridiculous political fighting under the guise of bureaucracy.

None of this means what she did was ok, but it's also hard to not look askance at the relentless witchhunting when it's placed in that broader context.

Personally I've reached a point where I'm done caring on the topic. There doesn't seem to be any kind of smoking gun, just a lot of hemming and hawing. Normally I would care about this, but honestly I'm a bit inured at this point. Where is the show of her using these specifically to avoid FOIA on work material actually relevant to FOIA?

That's really the only true relevant question when it comes to moving to private servers. Classified material isn't supposed to be on unclassified government servers either, so the attempt to focus on that (mostly with retroactive or improperly labeled material and a few other issues) really seems awkward when we're supposed to care about the private servers as if they're damning.

Most interesting to me was confirmation that the server was breached. Unknown parties accessed it from TOR multiple times.

From your link, an individual email account on the server was breached.

This happens all the time, for varying reasons, mostly due to a phishing compromise of the account, and occasionally due to password re-use and related vectors of compromise. While it's bad for the individual account's contents, it's absolutely irrelevant beyond that.

If that's the worst they can find then personally I'm actually impressed. I was expecting that the server(s) had been root/fully compromised at least once, given how they get perennially described. If that turns out to not be the case, then they've actually been run better and more securely than the State Department's [at least non-classified] servers, from all reports.

Look, getting all up in arms over crap like that link is why people like me are no longer convinced there's anything here worth paying attention to. I'm actually willing to listen if there's some kind of smoking gun, but that's some petty bullshit right there.

Most interesting to me was confirmation that the server was breached. Unknown parties accessed it from TOR multiple times.

Not sure why you are being down voted on newly revealed information that seems to confirm that one of the servers email accounts was breached.

If you're down voting him, perhaps an explanation as to why?

Probably because we know DOJ email servers have also been breached. He's implying that her servers were less secure and somehow put information in harms way. History seems to show us that it wasn't at any more risk.

Yeah, but the FBI is saying there was no evidence that the server was hacked. And then we find out that one of the email accounts was accessed over the TOR network and the user of the email account had never heard of TOR much less used it to access email.

That seems like yet another skewing of the finding to put them in the best possible light. (EDIT: not saying she was or was not, but I would say that there was indicators that it was possibly compromised)

DOJ, OPM, Pentagon, doesnt have any relevance on if she was irresponsible for having this whole set up. That same article states they werent even able to confirm if TLS was ever enabled. And Why? Because Clinton/IT took steps to make sure it couldnt be found out before turning over the equipment.

Just to clarify, the move to a hosted solution - with requested encryption - was initiated after Clinton's tenure as Secretary of State (January 21, 2009 – February 1, 2013) was completed in February, 2013, and FOIA requests were no longer applicable as she was no longer a government employee.

I think that would depend on the scope of the migration. Did they migrate all of the history over to the hosted solution? i.e. Did they migrate the OS, Exchange and BES servers into PRN's datacenter? Or, did they start from scratch with a clean slate, fresh install and no data migration. If it's the former and not the latter, I'd be pretty damned certain it'd still be subject to FOIA requests.

Agreed, my point was about future communications and not past data (post updated for clarification).

I read this article to say that Powell did the same thing decades ago and the only difference between now and then is that he didn't talk about.

That doesn't make it OK and he should be under investigation as well.

haven't you heard the law doesn't apply to republicans.

They were no laws broken by clinton than we can tell, it's just a weird thing. Powell clearly used private email to skirt records requests (and IIRC the Bush admin lost millions of emails). But Clinton seemed aware information is public record no matter how it's sent.

And if we compare the number of times this server was breached to government breaches, i don't know if this makes the idea of using your own server look like a bad idea. most intrusions are via social engineering, and there's probably a lot more weak points in the staff of gov email than this private one.

What i find strange is that Clinton was secretary of state, and was probably handling classified information constantly. How is it after the FBI has reviewed 45,000 of the 60,0000 emails there are so few classified emails being sent around (only 1 was sent BY clinton). Does the government just not send classified information through email at all? I'm more interested, from a technological perspective, in how this is handled.

Most interesting to me was confirmation that the server was breached. Unknown parties accessed it from TOR multiple times.

Not sure why you are being down voted on newly revealed information that seems to confirm that one of the servers email accounts was breached.

If you're down voting him, perhaps an explanation as to why?

Probably because we know DOJ email servers have also been breached. He's implying that her servers were less secure and somehow put information in harms way. History seems to show us that it wasn't at any more risk.

I didn't imply that at all. Here we have fairly solid evidence that a breach of Hillary's server happened. That seems to contradict the FBI's stance, Comey's statement and testimony, and is a first as far as I know.

And in comparison, the DOJs non-classified email systems were hacked. There is no evidence that the classified system ever was.

A 'breach' of an account is not a breach of the server. The account being access via TOR implies the user credentials were acquired through some means. Was this 'breached' account a classified account?

None of this means what she did was ok, but it's also hard to not look askance at the relentless witchhunting when it's placed in that broader context.

...

My personal evolution on this issue has gone from "having a privately controlled email server sounds really really bad, and was probably done to avoid monitoring! I'm really upset about this!" to "wow, these allegations sound extremely serious!" to "oh, those allegations were not really true at all" to "yikes, this again? how much more whining and knashing of the the teeth am I going to have to put up with?" If this had been any other politican, like, literally any other politician would we have heard more than a week or two about it? Would we have the FBI releasing their investigation documents to the public? Would all of Clinton's emails been open to the public like this? The amount of transparency, the lack of smoking guns, and the irrationally emotional anger have made me completely turn around on this issue.

Articles like this one are redeeming because they are somewhat technically interesting. Thanks for the great coverage, yet again, Ars.

Most interesting to me was confirmation that the server was breached. Unknown parties accessed it from TOR multiple times.

"multiple times" is 3 times in this case, and it wasn't the server that was breached, it was 1 person's email.

Even if this person was clinton herself, we already know there was not much damaging information stored on this server. And considering this seems more like someone used a weak password or was phished, this is a vulnerability no matter what email provider you're using.

Most interesting to me was confirmation that the server was breached. Unknown parties accessed it from TOR multiple times.

Not sure why you are being down voted on newly revealed information that seems to confirm that one of the servers email accounts was breached.

If you're down voting him, perhaps an explanation as to why?

Probably because we know DOJ email servers have also been breached. He's implying that her servers were less secure and somehow put information in harms way. History seems to show us that it wasn't at any more risk.

I didn't imply that at all. Here we have fairly solid evidence that a breach of Hillary's server happened. That seems to contradict the FBI's stance, Comey's statement and testimony, and is a first as far as I know.

And in comparison, the DOJs non-classified email systems were hacked. There is no evidence that the classified system ever was.

A 'breach' of an account is not a breach of the server. The account being access via TOR implies the user credentials were acquired through some means. Was this 'breached' account a classified account?

I could be wrong, but I think that all classified emails from DoD and State have to go through SIPRNet.

If this was strictly respected, then Clinton's server should contain no classified information. In real-life, we saw that a few classified things went through her personal email system, so it wasn't fully respected, or some of the info was not yet classified.

Most interesting to me was confirmation that the server was breached. Unknown parties accessed it from TOR multiple times.

"multiple times" is 3 times in this case, and it wasn't the server that was breached, it was 1 person's email.

Even if this person was clinton herself, we already know there was not much damaging information stored on this server. And considering this seems more like someone used a weak password or was phished, this is a vulnerability no matter what email provider you're using.

We're going to get into this in a story I'm currently writing (probably for next week, so it's not a Friday newsdumpster move). But it's worth noting THE ENTIRETY OF THE STATE DEPARTMENT'S UNCLAS EMAIL SYSTEM WAS PWNED FOR OVER A YEAR. I'm sorry, did I type that in all-caps? Also, between Chelsea Manning/ Wikileaks and the repeated hacks of State, the White House, etc between 2009 and 2014, it is highly likely that everything short of the TS/SAP stuff (and even some of that) that Clinton touched was already breached.

This does not excuse Clinton and her staff's—I'm looking at you, Jake Sullivan—for the extreme error of passing Top Secret/ Special Access Program classified data back and forth over Blackberries and a non-governmental e-mail system. I would expect that Sullivan, at a minimum, will have his clearance revoked and he will not be getting a job as a national security adviser if Clinton wins the election. Or at least, I think that's a reasonable expectation.

None of this means what she did was ok, but it's also hard to not look askance at the relentless witchhunting when it's placed in that broader context.

...

My personal evolution on this issue has gone from "having a privately controlled email server sounds really really bad, and was probably done to avoid monitoring! I'm really upset about this!" to "wow, these allegations sound extremely serious!" to "oh, those allegations were not really true at all" to "yikes, this again? how much more whining and knashing of the the teeth am I going to have to put up with?" If this had been any other politican, like, literally any other politician would we have heard more than a week or two about it? Would we have the FBI releasing their investigation documents to the public? Would all of Clinton's emails been open to the public like this? The amount of transparency, the lack of smoking guns, and the irrationally emotional anger have made me completely turn around on this issue.

Articles like this one are redeeming because they are somewhat technically interesting. Thanks for the great coverage, yet again, Ars.

I'm in the same boat. I almost feel ashamed at how upset i felt after first hearing about this.

But a year later, literally ~50,000 clinton emails reviewed, which is the vast majority, and we only know of 1 email hillary sent herself that was classified? She received a few emails of classified info that weren't marked as such but she can't control this. It seems like her server wasn't hacked, her email itself wasn't hacked-- what's the track record for the doj email system?

Clearly, as secretary of state, who would be handling classified info constantly, there was some other means she used to handle this info, and did so in practically all relevant cases.

The amount of money and FBI resources and new cycles being spent on this is ridiculous.

I could be wrong, but I think that all classified emails from DoD and State have to go through SIPRNet.

If this was strictly respected, then Clinton's server should contain no classified information. In real-life, we saw that a few classified things went through her personal email system, so it wasn't fully respected, or some of the info was not yet classified.

Some State unclas email included classified material. Mostly because of coordination issues and timeliness-convenience—people don't check their SIPRNET mail that often.

Most interesting to me was confirmation that the server was breached. Unknown parties accessed it from TOR multiple times.

Not sure why you are being down voted on newly revealed information that seems to confirm that one of the servers email accounts was breached.

If you're down voting him, perhaps an explanation as to why?

Probably because we know DOJ email servers have also been breached. He's implying that her servers were less secure and somehow put information in harms way. History seems to show us that it wasn't at any more risk.

Yeah, but the FBI is saying there was no evidence that the server was hacked. And then we find out that one of the email accounts was accessed over the TOR network and the user of the email account had never heard of TOR much less used it to access email.

That seems like yet another skewing of the finding to put them in the best possible light. (EDIT: not saying she was or was not, but I would say that there was indicators that it was possibly compromised)

DOJ, OPM, Pentagon, doesnt have any relevance on if she was irresponsible for having this whole set up. That same article states they werent even able to confirm if TLS was ever enabled. And Why? Because Clinton/IT took steps to make sure it couldnt be found out before turning over the equipment.

You know, this level of twisting is why you and Rommel are not credible on the topic. You just come off sounding like a conspiracy nut when you can go from the article linked to "her servers got hacked."

Let's be clear: if there had been a full breach, there would have been no need to be accessing an individual account over Exchange via TOR. You could just grab the whole thing directly, instead. This is, if anything, evidence of a lack of a full breach, at least by whatever actor was accessing the particular account in question.

But, you know, why don't you two just keep shooting yourselves in the kneecaps over this. It's not like your hyperbolic approach to this is hurting your credibility at all. We can either assume you're both excessively biased or incompetent on the topic from how you're running with that story.

Not that I'm calling you technically incompetent, mind. Unless you actually believe there's not a distinction between an email account being individually compromised and a "server being hacked." I expect you're just intentionally twisting what you're saying. But hey, maybe you don't actually know better?

The way you two are trying to play this is why you have so many people turning away in disgust—not at Hillary, but at the ongoing digging for gold and related hyperbole and even outright lies in what is more and more clearly a dustbowl, with the only apparent motivation being a smear campaign rather than anything to do with actual justice or a real care about security.