Ongoing exploits infecting tens of thousands of reputable sites running the Apache Web server have only grown more powerful and stealthy since Ars first reported on them four weeks ago. Researchers have now documented highly sophisticated features that make these exploits invisible without the use of special forensic detection methods.

Linux/Cdorked.A, as the backdoor has been dubbed, turns Apache-run websites into platforms that surreptitiously expose visitors to powerful malware attacks. According to a blog post published Friday by researchers from antivirus provider Eset, virtually all traces of the backdoor are stored in the shared memory of an infected server, making it extremely hard for administrators to know their machine has been hacked. This gives attackers a new and stealthy launchpad for client-side attacks included in Blackhole, a popular toolkit in the underground that exploits security bugs in Oracle's Java, Adobe's Flash and Reader, and dozens of other programs used by end users. There may be no way for typical server admins to know they're infected.

"Unless a person really has some deep-dive knowledge on the incident response team, the first thing they're going to do is kill the evidence," Cameron Camp, a security researcher at Eset North America, told Ars. "If you run a large hosting company you're not going to send a guy in who's going to do memory dumps, you're going to go on there with your standard tool sets and destroy the evidence."

Linux/Cdorked.A leaves no traces of compromised hosts on the hard drive other than its modified HTTP daemon binary. Its configuration is delivered by the attacker through obfuscated HTTP commands that aren't logged by normal Apache systems. All attacker-controlled data is encrypted. Those measures make it all but impossible for administrators to know anything is amiss unless they employ special methods to peer deep inside an infected machine. The backdoor analyzed by Eset was programmed to receive 70 different encrypted commands, a number that could give attackers fairly granular control. Attackers can invoke the commands by manipulating the URLs sent to an infected website.

"The thing is receiving commands," Camp said. "That means that suddenly you have a new vector that is difficult to detect but is receiving commands. Blackhole is a tricky piece of malware anyway. Now suddenly you have a slick delivery method."

In addition to hiding evidence in memory, the backdoor is programmed to mask its malicious behavior in other ways. End users who request addresses that contain "adm," "webmaster," "support," and similar words often used to denote special administrator webpages aren't exposed to the client exploits. Also, to make detection harder, users who have previously been attacked are not exposed in the future.

The malicious Apache modules are proving difficult to disinfect. Many of the modules take control of the secure shell mechanism that legitimate administrators use to make technical changes and update content to a site. That means attackers often regain control of machines that are only partially disinfected. The larger problem, of course, is that the highly sophisticated behavior of the infections makes them extremely hard to detect.

Eset researchers have released a tool that can be used by administrators who suspect their machine is infected with Linux/Cdorked.A. The free python script examines the shared memory of a sever running Apache and looks for commands issued by the stealthy backdoor. Eset's cloud-based Livegrid system has already detected hundreds of servers that are infected. Because Livegrid works only with a small percentage of machines on the Internet, the number of compromised Apache servers is presumed to be much higher.

Article updated to add detail about http daemon in the fourth paragraph.

Of course, I don't know, but I would guess that writing, "I would think a changed binary would be a noticeable difference because ..." would not have caused anyone to object.

If you're running a production system that is exposed to the general internet, you should be running tripwire or something similar that scans for modified binaries. In addition, you should be storing those file checksums on a separate system so that any compromise doesn't modify them as well.

I would think "lying to the inspector" would be a valid form of attack?

The issue here is cPanel, which doesn't use a packaging system to install Apache, hence the standard rpm -v to check for changes won't work in this case. You would have to have manually obtained the checksum from a known good baseline and use some third party tool to monitor for changes. While larger hosting providers may be doing this, I suspect many of the smaller ones are not.

This is a classic example of hindsight always being 20/20. What may seem obvious, now that the details are known, may not have been all that obvious beforehand. I also think there's a lot of whining because Dan didn't spell out X or Y. He has the burden of having to write for a large audience; he brings awareness of the issue which is incredibly important. For those who could be impacted, he provides plenty of reference links from which suitable research can be conducted. Hopefully the majority of server admins will do so. For those looking to Dan to provide exact details on how to fix your servers, I feel sorry for your customers.

75 Reader Comments

Linux/Cdorked.A leaves no traces of the hosts it has infected on hard drives. Its configuration is delivered by the attacker through obfuscated HTTP commands that aren't logged by normal Apache systems.

Someone may want to elaborate on this. I doubt one gains control just by making stuff up on the spot.

Malware that propagates through social engineering and is blatantly obvious like "AV Defender Pro 2017" or DDoS attacks or whatever are tame enough.

This kind of attack is smart, it's not flashy, or obvious. It just does what it needs to do, and it proactively conceals itself. It's what you don't realize that's attacking you that's most dangerous. Stuff like this terrifies me. Give me ransomware any day.

Yep scary and will only get worse from here. Modern code is just too complicated to secure properly, unless the entire dev team are security-conscious ninja-skill developers, and they aren't. Time will come when grave exploits are found in everything, not just OSs and computer apps but toasters and car computers and airplanes and factories and power grid and everything in between, because people keep connecting everything to the Internet . Hacking anyone and anything will become possible, just like bad 80s SciFi. ( <3 Max Headroom! )

Not clear from the article is this is an Apache vulnerability or if it's some plugin?

Researchers from a variety of firms have been able to detect the infections, but they still don't agree on exactly what's happening to cause them. From the article:

It also remains unclear exactly how legitimate websites are coming under the spell of the malicious plugins. While researchers from Securi speculate it takes hold after attackers brute-force the secure-shell access used by administrators, a researcher from Cisco Systems said he found evidence that vulnerable configurations of the Plesk control panel are being exploited to spread Darkleech.

really would prefer more info on the exploit vectors used to compromise the servers in question - I care a lot more about "this is how to secure your system" than "ooooooh, there's malware out there..."

Wouldn't a lot of these be mitigated by creating ACLs/firewall rules that didn't allow new outbound TCP connections? Public web servers need the public to be able to reach them, but they often don't need to establish new public connections themselves (at least not to arbitrary addresses).

really would prefer more info on the exploit vectors used to compromise the servers in question - I care a lot more about "this is how to secure your system" than "ooooooh, there's malware out there..."

Yes, it would be nice if that information was available. Unfortunately, it's not, and that's partly what makes this ongoing attack so important for people to know about.

Yep scary and will only get worse from here. Modern code is just too complicated to secure properly, unless the entire dev team are security-conscious ninja-skill developers, and they aren't. Time will come when grave exploits are found in everything, not just OSs and computer apps but toasters and car computers and airplanes and factories and power grid and everything in between, because people keep connecting everything to the Internet . Hacking anyone and anything will become possible, just like bad 80s SciFi. ( <3 Max Headroom! )

Linux/Cdorked.A leaves no traces of the hosts it has infected on hard drives. Its configuration is delivered by the attacker through obfuscated HTTP commands that aren't logged by normal Apache systems.

Someone may want to elaborate on this. I doubt one gains control just by making stuff up on the spot.

As usual, Dan has overly sensationalized things by leaving out important data. He did link to articles which provide it though.

The apache httpd binary itself is the infected file. It's the configuration parameters that live in shared memory (as they have to be, so they can be viewed by all httpd processes).

As for how it gets on the system in the first place, some of the known infections came from brute-forcing guessed user accounts via ssh (e.g. root/changeme). But the infection vector is unknown in many cases.

Linux/Cdorked.A leaves no traces of the hosts it has infected on hard drives. Its configuration is delivered by the attacker through obfuscated HTTP commands that aren't logged by normal Apache systems.

Someone may want to elaborate on this. I doubt one gains control just by making stuff up on the spot.

As usual, Dan has overly sensationalized things by leaving out important data. He did link to articles which provide it though.

The apache httpd binary itself is the infected file. It's the configuration parameters that live in shared memory (as they have to be, so they can be viewed by all httpd processes).

As for how it gets on the system in the first place, some of the known infections came from brute-forcing guessed user accounts via ssh (e.g. root/changeme). But the infection vector is unknown in many cases.

To be clear, my article /does/ note that brute-forcing user accounts is one possible vector, so I think it's not fair for your comment to suggest that detail was left out. Second, it's not entirely clear that all infections are caused by the Apache httpd binary itself. There is most definitely rogue plug-in modules used in many of the attacks. Do you have any insight what the relationship is between these plug-ins and the httpd?

Yep scary and will only get worse from here. Modern code is just too complicated to secure properly, unless the entire dev team are security-conscious ninja-skill developers, and they aren't. Time will come when grave exploits are found in everything, not just OSs and computer apps but toasters and car computers and airplanes and factories and power grid and everything in between, because people keep connecting everything to the Internet . Hacking anyone and anything will become possible, just like bad 80s SciFi. ( <3 Max Headroom! )

The problem isn't modern code, it's the old code modern code depends on. If you use a strict, granular permissions system from the kernel up, inevitable bugs are much harder to exploit. I look at security now kind of like I looked at multitasking in the Windows 9x days: it's a kludge, built on another kludge, so that other kludges can keep working, and just one failure in just one of those kludges will bring everything down.

One day we'll abandon the old APIs and laugh at how useless they are just as we can laugh at how it was possible for a pointer bug to overwrite the IDT in Windows 98. And it will be the so-called complex modern code that fixes the problems.

"Linux/Cdorked.A leaves no traces of the hosts it has infected on hard drives."

This leaves out an important detail, which the linked blog includes:

"The backdoor leaves no traces of compromised hosts on the hard drive other than its modified httpd binary"

Leaving out the second clause makes the infection seem advanced and bizarre. There are ways of storing malware without putting it on a disk, and your wording implies that this malware is capable of such.

My note about the infection vector was in response to other comments. You did include that in your article, and I did not mean to imply otherwise.

"Linux/Cdorked.A leaves no traces of the hosts it has infected on hard drives."

This leaves out an important detail, which the linked blog includes:

"The backdoor leaves no traces of compromised hosts on the hard drive other than its modified httpd binary"

Leaving out the second clause makes the infection seem advanced and bizarre. There are ways of storing malware without putting it on a disk, and your wording implies that this malware is capable of such.

My note about the infection vector was in response to other comments. You did include that in your article, and I did not mean to imply otherwise.

Fair enough. For the record, I had no intention of exaggerating the importance of these ongoing attacks. The ability to use unknown vectors to penetrate 20,000+ Apache-driven websites that surreptitiously infect end users with Blackhole malware is reason enough for this story to be published. I didn't include the detail about the httpd binary because I didn't consider it to be important. It's fine that you disagree with me. My only problem is your assumption that my reason for leaving out the detail was to sensationalize.

I suspect that the issue many folks have with the reporting in this article can be traced to the fact that, normally, an Ars article about malware includes detailed information about what platforms/applications it affects, how it spreads, what it does, etc.

That information is understandably lacking because it's not yet well known, if at all. Yet, the need to write about it is clear because the malware is, apparently, spreading quickly.

As a result, I think if the article had placed more emphasis and elaborated further on the details that are not yet known, along with clarifying how that adds to the danger, Dan would find it less necessary to clarify the current situation in the comments.

I suspect that the issue many folks have with the reporting in this article can be traced to the fact that, normally, an Ars article about malware includes detailed information about what platforms/applications it affects, how it spreads, what it does, etc.

That information is understandably lacking because it's not yet well known, if at all. Yet, the need to write about it is clear because the malware is, apparently, spreading quickly.

As a result, I think if the article had placed more emphasis and elaborated further on the details that are not yet known, along with clarifying how that adds to the danger, Dan would find it less necessary to clarify the current situation in the comments.

Thankfully, Dan does actually clarify his writing in the comments, something not often seen on other news sites. This back and forth with the community is why I actually like reading Ars, though I do wish it would happen a tad more from time to time.

"Linux/Cdorked.A leaves no traces of the hosts it has infected on hard drives."

This leaves out an important detail, which the linked blog includes:

"The backdoor leaves no traces of compromised hosts on the hard drive other than its modified httpd binary"

Leaving out the second clause makes the infection seem advanced and bizarre. There are ways of storing malware without putting it on a disk, and your wording implies that this malware is capable of such.

My note about the infection vector was in response to other comments. You did include that in your article, and I did not mean to imply otherwise.

Fair enough. […] I didn't include the detail about the httpd binary because I didn't consider it to be important.

It's reasonable to object to an unfounded accusation of sensationalism, but you were wrong to not consider that important, it's really important. Any binary modification on a drive presents a number of potential quickly implemented measures to at least provide detection, including tripwires, immutability and so on. From the article, it sounded like it might be some sort of purely regenerative attack. If that's not the case then yeah, that's significant.

The Python file linked from the article fails with invalid syntax on the octal constant on line 34. (Tested on RHEL 4 and CentOS 5.) It is a funny-looking constant. 0666 is 0x1B6, which does not remind me of any magic numbers.

The Python file linked from the article fails with invalid syntax on the octal constant on line 34. (Tested on RHEL 4 and CentOS 5.) It is a funny-looking constant. 0666 is 0x1B6, which does not remind me of any magic numbers.

The Python file linked from the article fails with invalid syntax on the octal constant on line 34. (Tested on RHEL 4 and CentOS 5.) It is a funny-looking constant. 0666 is 0x1B6, which does not remind me of any magic numbers.

Nope, 0666 is just a completely random nonsensical number

Sorry; I'm an old guy, and used to writing octal with a leading zero. The Python file (correctly) says, 0o666.

Some people would find mystical significance in the number 666, It's the "flags" field in the shmget call, so, 110110110, It's s'posed to be an int, so there's plenty of room. Dunno...

Edit: I'm a dullard, and literal-minded. I missed the wink. But I'd still like that sucker to work.

Seriously disappointed with the coverage of this issue, Ars is just the latest article I've had problems with.

Any filesystem level modifications required for this are a massively important and, even better, readily trackable. A Production web server should have it's binaries and configuration files tracked. It's not difficult, and done properly could be used to shut this vector down, even while it's specifics are investigated and tracked.

Only an idiot would think that any on disk modification of any binaries or modules is almost "invisible" as much of the hype has entertained. It's a disgrace actually.

Wouldn't a lot of these be mitigated by creating ACLs/firewall rules that didn't allow new outbound TCP connections? Public web servers need the public to be able to reach them, but they often don't need to establish new public connections themselves (at least not to arbitrary addresses).

The server is not making any outbound connections. Malicious modules will inject malware into the normal server responses back to the client.

0666 is of course also rw-rw-rw-. Make it world-read write, must be a virus :-)

I buy that. The shmget call takes a couple of params followed by standard Unix permission bits. What I don't understand is why that constant generates a syntax error. (Maybe there's something else I don't understand; I don't really speak Python, but line 34 generates a syntax error and there's a pointer aimed at the constant.)

Seriously disappointed with the coverage of this issue, Ars is just the latest article I've had problems with.

Any filesystem level modifications required for this are a massively important and, even better, readily trackable. A Production web server should have it's binaries and configuration files tracked. It's not difficult, and done properly could be used to shut this vector down, even while it's specifics are investigated and tracked.

Only an idiot would think that any on disk modification of any binaries or modules is almost "invisible" as much of the hype has entertained. It's a disgrace actually.

Exactly, I have alerts sent to me when system file checksums change.. dunno bout you guys, have fun with that.

Oh and Dan.. yea, the apache2 binary changing is a HUGE deal, just saying.

Seriously disappointed with the coverage of this issue, Ars is just the latest article I've had problems with.

Any filesystem level modifications required for this are a massively important and, even better, readily trackable. A Production web server should have it's binaries and configuration files tracked. It's not difficult, and done properly could be used to shut this vector down, even while it's specifics are investigated and tracked.

Only an idiot would think that any on disk modification of any binaries or modules is almost "invisible" as much of the hype has entertained. It's a disgrace actually.

I'm not sure why you think it's OK to refer to people as idiots. It's not. Please be more civil in this forum.

He probably would, but it still makes you sound like a pretentious asshat. To each his own though.

A production web server which is not being tracked for filesystem changes of binaries and configuration files is run by an incompetent administrator. It's not speculation, it's fact.

While such a web server may in fact be vulnerable to these modifications (the vector has yet to be described), such a modification should trigger an appropriate notification event that the administrator is aware of it immediately.

This article, and the ones it is based on, have all posited that filesystem level changes are nearly invisible; that can't be further from the truth. There are a number of tools which exist specifically for tracking such changes to production servers. Only an incompetent administrator would not use such tools on a production level web server.

Writing articles describe how "invisible" filesystem level changes are, and how difficult this particular malware infestation is nothing more that hyperbolic tripe, or utter incompetence.

Wouldn't a lot of these be mitigated by creating ACLs/firewall rules that didn't allow new outbound TCP connections? Public web servers need the public to be able to reach them, but they often don't need to establish new public connections themselves (at least not to arbitrary addresses).

The server is not making any outbound connections. Malicious modules will inject malware into the normal server responses back to the client.

Ok, sure. I was thinking more about the rash of "server-grade DDOS" that has been going around.