Webserver attack deconstructed

The emergency

Some days ago I got a call.

A webserver (Linux) of an advertising company hosting about 60 sites was going nuts and was driving attacks against other sites.
The datacenter had taken it offline twice and now demanded a reinstallation.

I asked for more information from the datacenter to get some overview of the situation and got some logs with tons of lines like this:

The frequency of this requests were about 100/s according to the logs.
This attack pattern could theoretically also originate from a programming error but most probably the server had been compromised.

As the host was taken offline and there was no way to access it by ssh I got the credentials for HP iLO on this server.
There I set up a firewall that blocked DNS requests that exceeded some 100 / second.
This was only because of paranoia – I didn’t plan to boot the server into the hacked system but who knows …

Then I had to convince the support staff of the datacenter to put the server back online without doing a re-installation.
My argument was that if there was a website with malicious code and we restored the backup the malicious code would again be in place. They agreed to put the server online in rescue mode.

Rescue mode

Usually this means that the host is booting via network. The OS resides in memory and the local disks can be mounted. You can even jump into the local OS using chroot.

Cleaning up

I had to find the cause for the DNS attack so I started looking for malicious code using some filescanners for it.

What I found was some index.html files that contained malicious javascript code coming from an attack called RunForestRun. This was installed using a vulnerability in Plesk Panel.

The javascript looks like this

I cleaned those files. Looking at the installed Plesk version I found out that it wasn’t prone to this attack anymore and I remembered that the client had told me that he moved his sites to the new host some time ago. So most obviously he moved the malware with it.

Could this really be the cause for the UDP flood? I digged deeper into RunForestRun and found out that ir was redirecting users accessing a website with this malware. But DNS flooding? No there seemed to be more going on.

Digging deeper

So what sites were running on this hos? Mostly wordpress.

So I started researching for known exploits for wordpress and I found that some are using malicious error pages, like 403.php – oviously because they are contained with many wordpress themes.