Security researchers have uncovered an ongoing and widespread attack that causes sites running three of the Internet's most popular Web servers to push potent malware exploits on visitors.

Linux/Cdorked.A, as the malicious backdoor behind the attacks is known, has been observed infecting at least 400 Web servers, 50 of them from the Alexa top 100,000 ranking, researchers from antivirus provider Eset said. The backdoor infects sites running the Apache, nginx, and Lighttpd Web servers and has already exposed almost 100,000 end users running Eset software to attack (the AV apps protect them from infection). Because Eset sees only a small percentage of overall Internet users, the actual number of people affected is presumed to be much higher.

"This is the first time I've seen an attack that will actually target different Web servers, meaning the attacker is willing to create the backdoor for Apache, Lightttp, and nginx," Pierre-Marc Bureau, Eset's security intelligence program manager, told Ars. "Somebody is running an operation that can victimize various Web servers and in my opinion this is the first time that has ever happened. This is a stealthy, sophisticated, and streamlined distribution mechanism for getting malware on end users' computers."

Previously, Cdorked was known to infect only sites that ran on Apache, which remains by far the Internet's most popular Web server application. According to this month's server survey from Netcraft, Apache and nginx are the No. 1 and No. 3 packages respectively, with about 53 percent and 16 percent of websites. The survey didn't rank Lighttpd, a Web server designed for speed-critical sites that's used by sites including Meebo, YouTube, and Wikimedia, according to Wikipedia. The report of the susceptibility of nginx came as its maintainers issued an update that patches a remote-code execution vulnerability in the open-source Web server. (There's no evidence the vulnerability is related to the Cdorked infection.)

Linux/Cdorked.A is one of at least two backdoors recently observed causing trusted and often popular websites to push exploits that attempt to surreptitiously install malware on visitors' computers. Like Darkleech, a backdoor estimated to have infected 20,000 Apache websites, it redirects users to a series of third-party sites that host malicious code from the Blackhole exploit kit. A recent blog post from security firm Invincea reports another rash of website hijackings, but they appear to be unrelated to Cdorked, and there's no indication Darkleech is involved, either.

Also similar to Darkleech, the Cdorked backdoor makes it extremely difficult for end users and even security researchers to notice their computers are being attacked. Users who speak Russian, Ukrainian, and at least four other languages are never exposed, and people who have already been attacked in recent days are also spared. Common configurations include a large list of IP addresses that are also blocked from exploits.

"We believe the operators behind this malware campaign are making significant efforts to keep their operation under the radar and to hinder monitoring efforts as much as possible," Eset researcher Marc-Etienne M.Léveillé wrote in a blog post published Tuesday. "For them, not being detected seems to be a priority over infecting as many victims as possible."

Cdorked-infected servers are also advanced enough to distinguish among different computing platforms used by end users visiting infected sites. Those using Windows machines are directed to sites that mostly host exploits from Blackhole. People using Apple iPads or iPhones are redirected to porn sites that may also be hosting malicious code. Cdorked also stores most of its inner workings in a server's shared memory, making it hard for some admins to know their sites are infected. Compromised systems can receive up to 70 different encrypted commands, a number that gives attackers fairly granular control that can be remotely and stealthily invoked.

In another testament to the ambition of its operators, Cdorked relies on compromised domain name system servers to resolve the IP addresses of redirected sites. The use of "trojanized DNS server binaries" adds another layer of obscurity to the attacks, since they make it easier for attackers to serve different sites to different end users.

"They are using the compromised DNS server to very accurately filter out who is going to visit the next stage Web server," Bureau said in an interview. "This means, for example, that security researchers will have a very hard time being served the same content as a victim. It makes the investigation and tracking this operation harder. They are trying to control every step along the way to make every visit very traceable but also very hard to recreate."

Researchers still don't know how servers are being infected with Cdorked. Because compromised machines are running a variety of administration controls, cPanel and competing software aren't obvious suspects. Cdorked doesn't have the ability to spread by itself and doesn't exploit a vulnerability in any other specific piece of software, either.

Readers who want to ensure their websites aren't infected should use the rpm –verify command to see if the HTTP daemon they use has been altered. Eset researchers have also released this free python script (zip file) to examine a server's shared memory for signs it is under the control of Cdorked.

Bureau said he believes Cdorked and Darkleech are two competing toolkits for exploiting Web servers. Their prevalence, combined with Invincea's discovery of popular websites also exposing visitors to malware attacks, suggests exploits are expanding beyond the traditional base of machines running Microsoft-based software.

"A couple years ago malware against the Linux operating system was really in the age of its proof of concept," he said. "Whenever we would discover something everybody would say: 'It's not really in wild. It's just somebody trying to prove a point.' Now the fact that we see so many instances of infected Web servers out there really shows we're past the era of the proof of concept. Now serious operators are making serious money by victimizing these web servers."

Story updated to add detail in the fourth paragraph that there's no evidence nginx vulnerability is related to Cdorked infections.

Promoted Comments

What sites are some of the major ones being affected?I see a note stating that 50 of the top sites are impacted... but are we talking ones that people here might regularly visit? And is the malware being pushed through ads, or delivered some other way? What kind of impact on the visitor are we looking at?

I'm finding it a little difficult (and it might just be the way I'm reading it) to separate the impact between the site owner and the site end user.

They are replacing the web server program with a malicious one. When a client requests a page they are in control over what happens. When they decide that they want to try to infect a visitor, they will send them a redirect instead of the actual page. They can also decide to act normally and not redirect. This is what is allowing them to hide from detection.

I'd be interested to know if BSD servers are also susceptible to infection and if FreeBSD jails may be able to play a part in quarantining infections to individual sites on shared hosts.

In theory, of course BSD servers can be attacked in the same way, if not with this specific attack; that said, Cdorked is a payload, not the actual exploit which gave the attacker root access in the first place.

We still have no idea how they attackers are getting their initial unprivileged access (hiya, some shitty php thing everyone runs!), nor how they elevate privs to root after the initial hack. I wouldn't quite call cdorked a payload, it's more of a complex payload delivery system... But on the server side we're basically talking about either modifying the web server daemon or loading a module that accomplishes the same. It's an open question then as to whether all the non-linux unixes are vulnerable - they probably share whatever application layer hole that exists, but they probably don't share the "www user can gain root" hole. There may be some other unknown root hole, but I'd say that's pretty unlikely. As for cdorked itself, we can only hope their source code is riddled with linuxisms that make it hard to compile on other unixes.

TL;DR, no, other unixes are probably not vulnerable, and no it's really, really unlikely the exact same root hole exists.

bersl2 wrote:

Not sure exactly what is possible with a FreeBSD jail, but there's almost always a way lurking around to break out of any sort of containment.

The jail would probably not mitigate the attack, but even with root it's unlikely the attacker would break out of the jail (many hosters give their VPS users root inside the jail, I would expect even more obscure breakouts would be known by now). Being jailed wouldn't keep you from getting infected, but I often run web servers inside a jail so that when (not if), some app is compromised, I can examine the jail from "outside" (ie: the host hosting the jails) where I know I have trusted binaries to examine the contents of the jail.

1) "People using Apple iPads or iPhones are redirected to porn sites that may also be hosting malicious code."But is the iPad itself getting sprayed with malware? This would be enormous news as there are not a lot of iOS viruses or other malware we know about. This needs deeper exploration.

2) Why is the kernel being blamed when it seems more likely that there is a simple Apache +/- plugin problem? Its like a headline saying OSX or Windows got pwned but then you read the fine print and its just another day and thus time for a new Adobe Flash or Adobe Reader exploit.

3) Why are these top 50 out of top 100,000 sites not getting outed? Avoiding going there or taking a morning after pill if you already did would be prudent right? Even if they are only the tip of the iceberg.

4) OMG corrupted DNS servers? Is that bad?-) I would be interested in more info about this angle. Clearly the server is at the host site but I do not get the relevance entirely. Surely I use my local ISP's DNS server? Or does it serve to mask the redirects from the compromised site so they don't realize whats going on? Or is it just a nice way to filter who goes where as the article states.

2) It also affects nginx & lighttpd so there are two possibilities: this uses multiple exploits to affect a variety of web daemons, or it is targetting part of the software stack that all these share in common like the kernel.

4) Using DNS servers merely adds another layer of obfuscation. It allows the attackers to serve multiple malware from a variety of locations, or arbitrarily not serve malware to some people, like those it suspects are honeypots. This makes it even harder to detect and track.

But what are the impacts and actions for the end users? Because the whole goal of this campaign isn't the web servers, it's the users who visit them...

The most important thing end users can do is to make sure that whatever platform they use is fully patched. This means the OS and all the apps running on it are fully updated. This especially includes Java, Reader and Flash for all three major OSes. The vast majority of the attacks included in Blackhole exploit already-patched vulnerabilities. Keeping your machine updated will protect you against all of these.

Unfortunately, patching your computer doesn't protect you against "zero-day" vulnerabilities, which by definition are exploited in the wild before there's a patch from the developer. This is the type of exploit that was recently used to target US nuclear researchers. The good news is that zero-day exploits aren't nearly as common as exploits that target already patched machines.

Noscript is good, but the problem with it is that it breaks so many websites that many users quickly disable it for a new site so they can read basic content on it. And since they're visiting a site they trust, they don't worry too much about doing this. As soon as they do, much of the protection offered by Noscript is gone.

Antivirus programs catch many but not all exploits, so that's another protection that should be considered mandatory for Windows users and optional/recommended for OS X and Linux.

The bottom line: There's no sure-fire way to block software exploits. The attacks reported in this story represent a significant threat precisely because they affect sites end users have reason to trust and that admins have no idea are infected.

What sites are some of the major ones being affected?I see a note stating that 50 of the top sites are impacted... but are we talking ones that people here might regularly visit? And is the malware being pushed through ads, or delivered some other way? What kind of impact on the visitor are we looking at?

I'm finding it a little difficult (and it might just be the way I'm reading it) to separate the impact between the site owner and the site end user.

What sites are some of the major ones being affected?I see a note stating that 50 of the top sites are impacted... but are we talking ones that people here might regularly visit? And is the malware being pushed through ads, or delivered some other way? What kind of impact on the visitor are we looking at?

I'm finding it a little difficult (and it might just be the way I'm reading it) to separate the impact between the site owner and the site end user.

They are replacing the web server program with a malicious one. When a client requests a page they are in control over what happens. When they decide that they want to try to infect a visitor, they will send them a redirect instead of the actual page. They can also decide to act normally and not redirect. This is what is allowing them to hide from detection.

What sites are some of the major ones being affected?I see a note stating that 50 of the top sites are impacted... but are we talking ones that people here might regularly visit? And is the malware being pushed through ads, or delivered some other way? What kind of impact on the visitor are we looking at?

I'm finding it a little difficult (and it might just be the way I'm reading it) to separate the impact between the site owner and the site end user.

Unfortunately, the Eset report doesn't identify any of the infected sites. There's no indication ads are involved in any way. Cdorked redirects users to sites hosting exploits from Blackhole, which is one of the nastier malware kits available, so it's reasonable to presume the threat to end users is serious.

For those that followed the Ars guide to setting up a web server, what's the equivalent to rpm -verify for Ubuntu's APT-based package management system?

debsums

Right--the debsums command, but as stated it'll only do something useful on packages containing an md5 sum.

According to the Nginx listserv post linked in Dan's piece, this vulnerability specifically affects Nginx 1.3.9 through 1.4.0. If you're on Ubuntu 12.04 LTS and are using the standard repos, the latest available version is 1.3.12 and is vulnerable.

If you're using the Nginx PPA, version 1.4.1 is already available and is not vulnerable. You can also grab the Nginx 1.4.1 or 1.5.0 source code, do a quick compile, and substitute in the new nginx binary over your existing one.

Reading comprehension fail for me--the vulnerability listed is *A* fix, but not the fix for the specific Cdorked hack.

From an end-user perspective, what would be a sign that a web server is infected? I assume that there would be somewhere a request going out to a malware server (.ru, presumably) where the malware is actually hosted, right? And that the malware is then installed via the execution of some script (whether it be js or ActiveX, or something similar)? As a result, would something like NoScript with a whitelist-only work to protect the end-user?

I'm assuming here that the malware is hosted on a separate server where the actual exploit resides, and that the compromised webserver only serves to redirect the user. But the article is not clear on whether that's the case.

From an end-user perspective, what would be a sign that a web server is infected? I assume that there would be somewhere a request going out to a malware server (.ru, presumably) where the malware is actually hosted, right? And that the malware is then installed via the execution of some script (whether it be js or ActiveX, or something similar)? As a result, would something like NoScript with a whitelist-only work to protect the end-user?

I'm assuming here that the malware is hosted on a separate server where the actual exploit resides, and that the compromised webserver only serves to redirect the user. But the article is not clear on whether that's the case.

As stated in the article, there are few if any signs to end users that the website they're visiting is infected. The whole way that Cdorked works makes the attacks extremely hard to investigate, so there's still quite a bit that researchers don't know. Yes, the malware is hosted on separate, third-party sites, that the Cdorked-infected sites will redirect to, under certain circumstances.

I'd be interested to know if BSD servers are also susceptible to infection and if FreeBSD jails may be able to play a part in quarantining infections to individual sites on shared hosts.

In theory, of course BSD servers can be attacked in the same way, if not with this specific attack; that said, Cdorked is a payload, not the actual exploit which gave the attacker root access in the first place.

Not sure exactly what is possible with a FreeBSD jail, but there's almost always a way lurking around to break out of any sort of containment.

All of the infected servers seem to be running linux. I would assume that researchers are now focusing on trying to find a zero day exploit in the linux kernel or some part of the software stack that most linux webservers share.

This malware seems so much more thought out than your run of the mill stuff. I am super suspicious that it was designed by a nation-state. While we have come to expect that sort malware to be a lot more targeted than this, there are plenty of conceivable reasons for a state to benefit from a more widespread attack like this one.

This malware seems so much more thought out than your run of the mill stuff. I am super suspicious that it was designed by a nation-state. While we have come to expect that sort malware to be a lot more targeted than this, there are plenty of conceivable reasons for a state to benefit from a more widespread attack like this one.

I'd agree, but then iOS devices are being redirected to porn sites? How would this help attackers stay below the radar? It's like someone built a space program, but they're joyriding the Crawler-Transporter around town.

I'd agree, but then iOS devices are being redirected to porn sites? How would this help attackers stay below the radar? It's like someone built a space program, but they're joyriding the Crawler-Transporter around town.

That is a good point. The only rational reason I can come up with for a nation-state to do that is to disguise the origin of the attack. Regardless it is pure conjecture on my part, but something about it does not feel right.

A couple years ago malware against the Linux operating system was really in the age of its proof of concept.

Our Linux Web servers were infected by a rootkit 12 years ago. It got installed through a DNS server vulnerability, the DNS servers happened to run on the same machines.

The hackers did nothing unusual for a few weeks, until new year's eve. At that point we lost control of the servers, so we had to go to the data center in the night between years, take out the hard drives, format them and reinstall everything from scratch, because we couldn't trust those machines anymore.

The malware was not in the "proof of concept" at all, it was very real.

However, the number of Windows XP/2000/98 vulnerabilities and attacks at that time made this kind of niche malware for Linux almost a myth, and not worth mentioning.

I'd be interested to know if BSD servers are also susceptible to infection and if FreeBSD jails may be able to play a part in quarantining infections to individual sites on shared hosts.

In theory, of course BSD servers can be attacked in the same way, if not with this specific attack; that said, Cdorked is a payload, not the actual exploit which gave the attacker root access in the first place.

We still have no idea how they attackers are getting their initial unprivileged access (hiya, some shitty php thing everyone runs!), nor how they elevate privs to root after the initial hack. I wouldn't quite call cdorked a payload, it's more of a complex payload delivery system... But on the server side we're basically talking about either modifying the web server daemon or loading a module that accomplishes the same. It's an open question then as to whether all the non-linux unixes are vulnerable - they probably share whatever application layer hole that exists, but they probably don't share the "www user can gain root" hole. There may be some other unknown root hole, but I'd say that's pretty unlikely. As for cdorked itself, we can only hope their source code is riddled with linuxisms that make it hard to compile on other unixes.

TL;DR, no, other unixes are probably not vulnerable, and no it's really, really unlikely the exact same root hole exists.

bersl2 wrote:

Not sure exactly what is possible with a FreeBSD jail, but there's almost always a way lurking around to break out of any sort of containment.

The jail would probably not mitigate the attack, but even with root it's unlikely the attacker would break out of the jail (many hosters give their VPS users root inside the jail, I would expect even more obscure breakouts would be known by now). Being jailed wouldn't keep you from getting infected, but I often run web servers inside a jail so that when (not if), some app is compromised, I can examine the jail from "outside" (ie: the host hosting the jails) where I know I have trusted binaries to examine the contents of the jail.

All of the infected servers seem to be running linux. I would assume that researchers are now focusing on trying to find a zero day exploit in the linux kernel or some part of the software stack that most linux webservers share.

It would be very interesting to see the version of the linux kernel each infected host is running. Then which distro to see if the root vuln is kernel-level or some userland fuck-up.

I would not be shocked if the servers involved were early adopters looking for some "super go fast" bleeding edge kernel changes. Sometimes performance-centric folks are kind of blind to security if the speed increase is really notable.

From what I have read, the initial compromise is through SSH brute force. Keys or strong passwords, everybody!

SSH credential cracking is one of many theories about what's causing servers to be infected with DarkLeech and Cdorked. As far as I'm aware, this theory hasn't been proven or debunked. In any case, brute force attacks are extremely rare. Assuming SSH credentials are being cracked, it's most likely by guessing commonly used passwords.

A couple years ago malware against the Linux operating system was really in the age of its proof of concept.

15 years ago most hacks were unix, mostly Linux and Solaris.

For an end user your best defense against Blackhole and similar: Patch everything on your computer, starting with browser attachments. Remove every software package you don't need, starting with browser attachments, so you have less patches and zero days to worry about.

1) "People using Apple iPads or iPhones are redirected to porn sites that may also be hosting malicious code."But is the iPad itself getting sprayed with malware? This would be enormous news as there are not a lot of iOS viruses or other malware we know about. This needs deeper exploration.

2) Why is the kernel being blamed when it seems more likely that there is a simple Apache +/- plugin problem? Its like a headline saying OSX or Windows got pwned but then you read the fine print and its just another day and thus time for a new Adobe Flash or Adobe Reader exploit.

3) Why are these top 50 out of top 100,000 sites not getting outed? Avoiding going there or taking a morning after pill if you already did would be prudent right? Even if they are only the tip of the iceberg.

4) OMG corrupted DNS servers? Is that bad?-) I would be interested in more info about this angle. Clearly the server is at the host site but I do not get the relevance entirely. Surely I use my local ISP's DNS server? Or does it serve to mask the redirects from the compromised site so they don't realize whats going on? Or is it just a nice way to filter who goes where as the article states.

All of the infected servers seem to be running linux. I would assume that researchers are now focusing on trying to find a zero day exploit in the linux kernel or some part of the software stack that most linux webservers share.

This malware seems so much more thought out than your run of the mill stuff. I am super suspicious that it was designed by a nation-state. While we have come to expect that sort malware to be a lot more targeted than this, there are plenty of conceivable reasons for a state to benefit from a more widespread attack like this one.

That was addressed in the article:

Quote:

Cdorked doesn't have the ability to spread by itself and doesn't exploit a vulnerability in any other specific piece of software, either.

So I am guessing the initial attack vector must be something else, maybe social engineering of admins, or social engineering to get it integrated into some other piece of trusted software or repo or whatever

"A couple years ago malware against the Linux operating system was really in the age of its proof of concept," he said. "Whenever we would discover something everybody would say: 'It's not really in wild. It's just somebody trying to prove a point.' Now the fact that we see so many instances of infected Web servers out there really shows we're past the era of the proof of concept. Now serious operators are making serious money by victimizing these web servers."

Cdorked-infected servers are also advanced enough to distinguish among different computing platforms used by end users visiting infected sites

Really? Is running the UA string through an if/then/else statement considered "advanced" here at Ars?

Think it through. Previously you got some specific malware no matter what platform you were on from a bad ad on some site. On a Mac or linux box? Here try this Windows malware.

Now it is advanced. On a Mac? Here go to the Mac exploit sites. On a PC? Off to the PC sites. On an iPad? No malware for you, but go check out our sponsors at these porn sites and click on something so we get cash back.

This is the definition of more advanced right? Multiple web servers, multiple platform exploits / money making schemes.

1) "People using Apple iPads or iPhones are redirected to porn sites that may also be hosting malicious code."But is the iPad itself getting sprayed with malware? This would be enormous news as there are not a lot of iOS viruses or other malware we know about. This needs deeper exploration.

2) Why is the kernel being blamed when it seems more likely that there is a simple Apache +/- plugin problem? Its like a headline saying OSX or Windows got pwned but then you read the fine print and its just another day and thus time for a new Adobe Flash or Adobe Reader exploit.

3) Why are these top 50 out of top 100,000 sites not getting outed? Avoiding going there or taking a morning after pill if you already did would be prudent right? Even if they are only the tip of the iceberg.

4) OMG corrupted DNS servers? Is that bad?-) I would be interested in more info about this angle. Clearly the server is at the host site but I do not get the relevance entirely. Surely I use my local ISP's DNS server? Or does it serve to mask the redirects from the compromised site so they don't realize whats going on? Or is it just a nice way to filter who goes where as the article states.

2) It also affects nginx & lighttpd so there are two possibilities: this uses multiple exploits to affect a variety of web daemons, or it is targetting part of the software stack that all these share in common like the kernel.

4) Using DNS servers merely adds another layer of obfuscation. It allows the attackers to serve multiple malware from a variety of locations, or arbitrarily not serve malware to some people, like those it suspects are honeypots. This makes it even harder to detect and track.

All of the infected servers seem to be running linux. I would assume that researchers are now focusing on trying to find a zero day exploit in the linux kernel or some part of the software stack that most linux webservers share.

This malware seems so much more thought out than your run of the mill stuff. I am super suspicious that it was designed by a nation-state. While we have come to expect that sort malware to be a lot more targeted than this, there are plenty of conceivable reasons for a state to benefit from a more widespread attack like this one.

That was addressed in the article:

Quote:

Cdorked doesn't have the ability to spread by itself and doesn't exploit a vulnerability in any other specific piece of software, either.

So I am guessing the initial attack vector must be something else, maybe social engineering of admins, or social engineering to get it integrated into some other piece of trusted software or repo or whatever

That does not address my question. I was not wondering if Cdorked spreads itself, it does not make sense that it would if they are trying to remain stealth. Instead, I am curious if it is a nation-state sponsored virus that is spreading Cdorked as a payload. In my opinion it is way too widespread to be a result of social engineering. More likely it is a self destructing virus or automated brute forcing of SSH passwords.

"A couple years ago malware against the Linux operating system was really in the age of its proof of concept."

The author hasn't definitively stated if the affected servers were Unix/Linux based only. Apache, Nginx, and lighttpd are all available on Windows as well. Since IIS wasn't mentioned it does seem to point to a Linux/Unix exploit.

I do not know if they would help or not but I have been running rkhunter and chkrootkit to help prevent from such problems. They need to be on the system before going live. Both programs are in the repos..

From what I have read, the initial compromise is through SSH brute force. Keys or strong passwords, everybody!

And disallow root log-on via ssh, and make sure iptables rejects more than 3 or so ssh connection attempts from one source within 30 minutes.

Our servers out in the wild can only be SSH'd into from one routable IP address. I know that spoofing an IP address is trivial, but it's better than leaving SSH open to the world. In fact, that SSH port (non-standard) on both our web and database servers in the wild are the only port open to any outside IP address. The db can only be accessed from the internal network, and the web servers sit behind a load balancer, so they only serve up HTTP/S over the internal network as well.

I feel fairly safe with a configuration like this, but I'm always looking to lock it down more, so any suggestions are more than welcome.