Share this story

A code-hosting service that boasted having a full recovery plan has abruptly closed after someone gained unauthorized access to its Amazon Web Service account and deleted most of the customer data there.

Wednesday's demise of Code Spaces is a cautionary tale, not just for services in the business of storing sensitive data, but also for end users who entrust their most valuable assets to such services. Within the span of 12 hours, the service experienced the permanent destruction of most Apache Subversion repositories and Elastic Block Store volumes and all of the service's virtual machines. With no way to restore the data, Code Spaces officials said they were winding down the operation and helping customers migrate any remaining data to other services.

"Code Spaces will not be able to operate beyond this point," a note left on the front page of codespaces.com said. "The cost of resolving this issue to date and the expected cost of refunding customers who have been left without the service they paid for will put Code Spaces in a[n] irreversible position both financially and in terms of on going credibility. As such at this point in time we have no alternative but to cease trading and concentrate on supporting our affected customers in exporting any remaining data they have left with us."

"Backing up data is one thing, but it is meaningless without a recovery plan, not only that [but also] a recovery plan—and one that is well-practiced and proven to work time and time again," the cache stated. "Code Spaces has a full recovery plan that has been proven to work and is, in fact, practiced."

Wednesday's advisory said the unauthorized access to the Amazon-hosted Code Spaces data came on the heels of a distributed denial-of-service attack, presumably in an attempt to extort money from the service. Code Spaces officials wrote:

Dear Customers,
On Tuesday the 17th of June 2014 we received a well orchestrated DDOS against our servers, this happens quite often and we normally overcome them in a way that is transparent to the Code Spaces community. On this occasion however the DDOS was just the start.

An unauthorised person who at this point is still unknown (All we can say is that we have no reason to think [it's] anyone who is or was employed with Code Spaces) had gained access to our Amazon EC2 control panel and had left a number of messages for us to contact them using a hotmail address

Reaching out to the address started a chain of events that revolved [around] the person trying to extort a large fee in order to resolve the DDOS.

Upon realisation that somebody had access to our control panel we started to investigate how access had been gained and what access that person had to the data in our systems, it became clear that so far no machine access had been achieved due to the intruder not having our Private Keys.

At this point we took action to take control back of our panel by changing passwords, however the intruder had prepared for this and had already created a number of backup logins to the panel and upon seeing us make the attempted recovery of the account he proceeded to randomly delete artifacts from the panel. We finally managed to get our panel access back but not before he had removed all EBS snapshots, S3 buckets, all AMI's, some EBS instances and several machine instances.

In summary, most of our data, backups, machine configurations and offsite backups were either partially or completely deleted.

This took place over a 12 hour period which I have condensed into this very brief explanation, which I will elaborate on more once we have managed our customers' needs.

The advisory didn't say exactly how the hacker gained entry to the Code Space panel hosted on AWS. People who host their services on Amazon should avail themselves of the full spectrum of multifactor protections.

Promoted Comments

Has AWS responded? For example, when the S3 buckets were deleted. Did AWS offer any sort of recovery? An S3 storage plan can have data migrated to Glacier, which provides a backup. If that was done, wouldn't that imply that AWS would have a backup sitting on a Glacier server of the S3 data?

No, It's up to YOU to set up backups, fail safes, Glacier, multi-factor authentication.

Amazon provides a set of tools, it's up to you to use them properly. These people apparently offered a backup and recovery service with a single point of failure, which is not a great design.

Can someone please explain a bit about how Amazon EC2 works because I don't get Code Spaces' logic. Their statement says an intruder got into the EC2 control panel and left them messages. Doesn't that mean an admin account was already compromised?

The AWS Control Panel is like physical access to your data center. Someone could at that point take a wrecking ball to each server. Or pick them up and take them.

You have the ability to delete disks, computers, redundancy sets, and other AWS services (en masse). And that is delete forever, not just move to the trash can. There are ways to grant only certain privs to certain admin accounts (like separate data centers, or cages in a single data center). But from the sounds of things, these guys got violated at the master account level (bad news for them and their customers).