So, from where I am sat right now, it looks like Orain is dead. It could just be me but Orain has been struggling with issues for a while now and the events that took place last week were basically the final nail in the coffin.

As final nails in coffins go, I don’t see this nail being removed any time soon.

Some brief history and facts

Orain was founded by Dusti and Kudu in July 2013, which means it is currently just over 2 years old. Orain has no paid ‘staff’ but instead is kept on-line by a small team of volunteers. The number of volunteers trying to keep the sites up at any given time has varied although in the past month of so that number dropped to 2. During the 2 years the Orain services have been on a variety of different hosts, including AWS, Ramnode and most recently DigitalOcean.

Last week ( 16 Sept 2015 )

Firstly I will say that I still do not know exactly what happened, or how, but it must have been one of the following things:

Someone did something stupid with a password. This could have been accidentally posting it somewhere, sharing it with someone or not keeping it in a secure location.

Someone with access to the accounts@orain.org email forwarder had their email address compromised.

Someone on the inside decided that it was time for Orain to die…

There are other options but frankly the likely hood of those compared with the list above is slim.

Rough Timeline (UTC)

01:44 the CloudFlare password was reset (we have a an IP address relating to the reset of this password).

At some point the mail DNS records for orain.org were changed, pointing to an external server (not in orain control).

At some point the password for DigitalOcean was reset, made easy by the fact that this person had control of the email accounts.

At some point 1 in 2 requests were redirected to a questionable device. You can find an image of the change that was made here.

09:20 I woke up to see Orain in a mess and Informed Dusti and others by email while trying to see what on earth happened.

16:00 Confirmed that the ATT databases was no longer on the server. A screenshot can be seen here.

Also confirmed that someone had root access to the servers using the DigitalOcean panel, screenshot can be seen here. (It should be noted this shows the root user as idle for 9 hours at 16:00 UTC, meaning at least for prod5 the user was active last at roughly 07:00.

At some point in the afternoon / evening all machines were powered down.

What I can say with 100% certainty.

I have backups from 15th June 2015 @ 18:00 UTC for all wikis that existed at that time and I am more than happy to give these to people.

The ATT database was deleted, but I was not able to SSH to the primary database server so those databases may not have been deleted.

As the user had root on all servers via the DigitalOcean control panel it should be assumed that ALL data was / could have been compromised. This includes usernames, email addresses, names and hashed & salted passwords. This also includes access logs meaning IP addresses, user agents and request data which could all be tied to users.

I do not have any backups of the uploads, although these had not been deleted before the machines were powered down.

Right now I have no idea if the machines were simply powered down or deleted (they are only VPSs after all)

At this time I believe Dusti is trying to gain access back to the DigitalOcean and Cloudflare accounts, until this happens it’s hard to really say or do anything more.

Possible conclusions to all of this

Orain gets access back to DO, the servers are still there, it is powered up and the dbs & uploads are still there

Orain gets access back to DO, the servers are still there, it is powered up and a mixture of dbs and uploads are still there.

Orain gets access back to DO, the servers are still there, it is powered up and all the dbs are gone & the uploads are gone.

Orain gets access back to DO, the servers are gone…

Orain does not get access back to DO…

EDIT (well, option 6 here happened.)

Finally

I am happy to answer any questions I can, although basically everything I can say is written above.

As I previously said I would have expected the founders of Orain to inform the users of Orain of the events, but apparently they haven’t found the time to, or don’t want to, or a mixture. I hope that they will soon.

Personally I want to make try to help everyone that did have a wiki with Orain, I have the backups and am of course willing to give them to the wiki owners so that they can move to new hosting.

During May of 2015 Orain was the target of a DDoS attack. The attack ended up lasting roughly 9 days and bringing the service to its knees repeatedly. The ‘official’ timeline of events and write up can be found here. Below I will discuss why the details of the DDoS as well as how it took Orain down so easily and the measures that have now been put in place.

Details of the DDoS

The DDoS was first detected on May 20th and immediately took down all of the Orain services, although at this stage no one knew that it was a DDoS, we simply thought Orain was having IPv4 routing issues as the site was still accessible on IPv6.

After messaging DigitalOcean support they revealed to us that they had nullrouted our load balancing instance (the main instance needed to access the website) due to an inbound DDoS, they also apologised for a lack of automated email about this.

The image to the right (sorry for the poor quality, apparently I didn’t take a screenshot but took a snap on my phone) shows the spike of around 800Mbps inbound on the public network interface and a small increase in the internal traffic. After this spike both interfaces can be seen to fall to 0Mbps. This was due to the public IPv4 address being nullrouted by DigitalOcean.

After a period of time DigitalOcean would return the original route of the IP to our box and service would be restored for a short period before the next round of the DDoS hit us, and repeat…….

The DDoS did not concentrate on a single instance but as all public IPs were available to the world on GitHub the attackers could easily target them all and bring down every last service Orain was running, mail, dns, web, stats etc.

What changes have been made

So as discussed above the reason the attack hit Orain so hard was because all of the public IPs for servers were available to be abused. This in combination with our services being run on VPSs which nullroute traffic when DDoSs are detected meant attacking Orain was really quite easy..

Firstly I made the switch to CloudFlare. This would mask the IP addresses of the servers when, for example, requesting meta.orain.org. Previously this would have pointed you directly to the public IP address of our load balancer, now it will direct you to an IP address for CloudFlare’s CDN. Of course with a change of name servers we had to wait for this change to propagate around the world.

This in itself was not enough as of course the attackers already had our IP addresses and continued to DDoS Orain once protected behind CloudFlare, thus we needed to rotate the IP addresses of all of our servers and make sure that the new IPs were no longer visible anywhere.

The easiest place to find all of our IP addresses was in our public DNS configuration repo located at https://github.com/Orain/dns. With the move to CloudFlare this repo would no longer be used, so check, no new public IPs here!

The second place that public IPs could be found was within our Ansible playbook located at https://github.com/Orain/ansible-playbook. All public IPs were replaced with private internal IPs in this change as well as the addition of a hosts file to ensure all instances always resolved Orain domains locally rather than being pointed to CloudFlare.

So, Lastly, IP rotation. DigitalOcean do not provide a user with an easy way to grab a new public IPv4 address for a machine as of course this is something that could be easily abused and thus the process takes a bit more time. Each box must be shut down, a snapshot created, then a new box created from the snapshot. For all of Orain’s servers this took me roughly 1 hour.

TADA! No more DDoS :)

Side effects of the changes

The Orain DNS configuration is no longer publicly accessible and contributing to it is of course more difficult. This isn’t something that really needs to change that much though.

CloudFlare only supports wildcard domain support with full proxy services if you pay them $5000 per month, which of course Orain can not do. Thus Orain now has a CNAME record for every wiki that it hosts. A change has also been made to our CreateWiki extension to automatically create these.

No SSHing or accessing servers directly via domains. Previous to this we could simply ssh to ‘prod10.orain.org’ for example, but now of course this domain points to a CloudFlare IP address.

Custom domains of course broke. Custom domains were either pointing at the old Orain name servers which are no longer being used, or at the IP address of the load balancer instance which has no changed. Currently as we want to keep our IP addresses secret the only way to keep your custom domain working is to CNAME it to ‘lb.orain.org’ which per an RFC may not actually be allowed (but works with multiple hots all the same). See this stackoverflow question. Another solution to this would be to have a second load balancer with a public IP not routed through CloudFlare, thus if it was ever DDoSed then only custom domains would see downtime.

Our traffic is now being routed through the CloudFlare CDN which has apparently saved 25% of the bandwidth from needing to go to our servers (around 3GB) according to their dashboard.

Digital Ocean is a ‘cloud’ hosting provider focusing on VPSs with SSDs called droplets.

They are generally rather cheap for what you are getting and their entry level droplet with 512MB memory, 1 core, 20GB disk and 1TB transfer only costs $5 per month ($0.007 per hour).

Orain, one of the projects that I am currently helping out with, currently powers all of its services with a handful of these droplets, and through the maintenance of these services I realise more and more that aspects of Orain really are not suited to live on DO.

Orain is a community-driven, not-for-profitwiki network that I help to maintain.

It runs Mediawiki and has been around for the past couple of years. Over the years it has been hosted on VPSs from multiple different providers and its technical layout has changed massively from each provider. Below I will try to summaries it’s current layout! This will include:

The machines / VPSs – ( how many there are, what they are doing and why )

MassAction is a Mediawiki extension allowing users to perform mass actions on targets on a Mediawiki site through a static page using Mediawikis inbuilt job queue that I have been working on for the past half a year or so. I look forward to releasing it to the open source world soon!

addwiki is a collection of Mediawiki related PHP libraries (including one for wikibase). Previous to this I developed various PHP scripts and bots for Wikipedia using other libraries and always found that they were quite badly coded and prone to doing unexpected things. Addwiki is the start of my attempt to fix that for PHP.

Orain (github) is a community-driven, not-for-profitwiki network that I help to keep running.