DigitalOcean locking down droplets

I’m having the time of my life today after Digital Ocean conveniently locking down one of my droplets today. The support was completely clueless and worse, some of them weren’t even reading the replies. They take 1-2hr between replies and apparently it’s always not the same guy so the next guy would just briefly skim through the thread and insert another canned reply.

I’m left with no access at all to the server, not even to backup the data and having to deal with the terrible people at customer support. Seriously, I’ve had better experience with 5$/mth shared hosting.

The latest one-line reply I get was asking me to completely destroy the droplet… Seriously?

The guy who locked down my droplet said that my droplet was sending out ddos attacks and accused that it was a vulnerability with the elasticsearch used on my server. Apparently since I didn’t even have access to my server logs there is no way I could verify whatever he said.

Elasticsearch has a flaw in its default configuration which makes it possible for any webpage to execute arbitrary code on visitors with Elasticsearch installed.

.. which is already a clear indication that this vulnerability on exists on DEVELOPERS machine, not production servers. As far as I know, ES was already configured for closed system operation only so it’s not possible to send DDOS attacks – unless its sending the packets to our app itself, which is very likely when you do a lot of searches.

I still have my account at MT, wouldn’t recommend moving away to DO if you want to save yourself the sufferings. Already wasted ~6 hours trying to communicate with the support.

I noticed that our websites were down yesterday. Our big team has a big droplet on DigitalOcean to host all of our websites…

I talked to our team and our friend who handle the server said that our droplet was suspended…

Okay, stay calm…

The first email said that DO got complaint from Bank of America said that one of our websites host phishing page of them…

The second email said that one or some of our websites become DDOS attack source, possibly because of elasticsearch vulnerability… We also do not believe that we have elasticsearch installed…

Then, we jump to conclusion that one of our websites was hacked! And the hacker uses our website to generate bad traffic for his bad purposes…

We keep communicating with them, and in our conversation we find that the phishing page is on one of our “old” WordPress demo website… This subdomain is no longer maintained, so it still use an old version of WordPress…

We know that old WordPress version is full of vulnerabilities… So, it can be a door for hacker to hack our website and use it for bad purposes…

Our dropplet was still suspended, but DO give access via console… Then our team can “remove” this subdomain to fix this issue…

And, finally… our websites are back online again last night before Netherland vs Argentina match…

Lesson Learned

1) DigitalOcean (and also Linode) is “Unmanaged” VPS hosting… So, we are fully responsible with what happened with our server… We are glad that we have member in our team that has good capability on sysadmin…

2) We need to deactivate / remove our unused / inactive websites…

3) We need to keep our WordPress up to date…

4) We need to keep our WordPress secure…

I am not sysadmin, so I am not the expert on this case, I only share what I heard and I knew… And I hope this reply is useful for you…

I’ve been managing VPS for quite some time from small to larger ones, but this is the first time I was locked down and asked to completely destroy everything just because they suspected vulnerabilities that they didn’t even care to investigate properly.

The droplet that was locked down was a completely closed system for an app with maximum 5 users, so it is very tiny. If such a small app gave me this much problems, I would definitely stay away from DO for any larger scale websites.

I can live with unmanaged VPS, but not with this type of terrible support.

It is common when a company has bigger customer base, they need more resources for support… For DO case, their support is less responsive lately, for sure…

I also can feel that “elastic search” email is a generic answer template from their support…

For my personal stuff, I moved from DO to Linode last month… Their price is competitive now… Their documentation / library is better (DO is community-driven documentation)... And, their support is better, I think…

Post Reply

<strong></strong> to make things bold
<em></em> to emphasize
<ul><li> or <ol><li> to make lists
<h3> or <h4> to make headings
<pre></pre> for code blocks
<code></code> for a few words of code
<a></a> for links
<img> to paste in an image (it'll need to be hosted somewhere else though)
<blockquote></blockquote> to quote somebody