08 May 2014
This is the analysis story based on the incident handling on the server side incident, caused by a hack to perform some malicious attack to a compromised server, so it is the server side malware analysis, with using the rather sophisticated method of LD_PRELOAD, with the summary as per below:

In the end of March 2014 I helped a friend who has problem with his service perimeter from a hack case. The attack was a classic WordPress hack using the vulnerability scanner on certain user's directory of ".../wp-content/uploads", which was set with the unrecommended UNIX permission on some layer of services by the users. The service was using FreeBSD with the Linux binaries compatibility environment installed, and the tools used for this attack is meant to aim the both platforms.

The attack was successfully exploiting the shell and the attacker was compromising the server's wordpress related user's privilege by injecting the WSO shell and through it we have some malicious PHP scripts which the attacker was using it to perform attack to other services, dropping malicious binaries and implementing the "patch" to the system using the LD_PRELOAD scheme to execute the malicious binary as the backdoor to some services.

I was lucky can perform the overall analysis for this incident and this is the share of what was investigated in a chronological basis which hoping the information can be used for prevention and mitigation for the further attacks. All of the holes was covered, incident was mitigated and followed well and now I can write (with permission) for a share.

A flash Update1: Recent Analysis of Incident between July 25 to August 5th 2014 is compiled in Kernel Mode posts

Answering some requests about this post due to the threat, I was asked to hint a how-to detect the global wordpress sites infected by this malware. The good news: It is searchable. Bad news is: Tons of them now.

So I made the video of based on list on the CURRENT infected sites we detected so far (is not all infection, you can help to add the list). Below is the video:

The infected sites list is here-->>[MMD Pastebin] and the "recent" libworker.so downloadable URL list is here -->>[MMD Pastebin] < There are a lot more of these, and mostly the installer of malware library are dated from March 2014. The clean up also operations are in progress too, please bear for the 404 that might occur in some sites. You can use the list as the hint to detect the rest of infection in world wide web.

The malware and its components (PHP script for dropper. installer and companion malicious scripts) were uploaded from remote via WSO PHP Shell. In all infected cases I found the WSO up and alive, WSO shell script is obfuscated under camouflaged common PHP filename so is a bit hard to spot which one, below is the video PoC of this statement:

The Compromised Wordpress

The version is recent, I was expecting older version but it is not the case, and I confirmed it myself. Since I am not using wordpress even though handling many incident of WordPress hacks, I am actually still wondering why it has the permission #777 on "wp-content/uploads" directory which was suspected to be the only "gate" for the attack to get in. Maybe is the user settings? I just can't be sure..

Anyway, using the same methodology to search for the injected codes in WordPress as per described in here -->[ 1 ] [ 2 ] and the malicious files injected were fetched successfully as per below snapshot:

Some "accompanied" malicious injected codes

Based on the attack log, history and time stamped we know that the checkandall.php is the first file injected into the system, and contains the below code:

Obviously this is the script file used by the attacker to check the overall environment of the entered directory and its files after compromising, the value of $all_files will be the contents parsed in an array, as per shown in below illustration:

The next files are the point of this post, the nextstyles.php, oldstyle.php and stylered.php, which are files coded with a one-liner PHP obfuscation in eval/gzuncompress/base64 php code as per illustrated below:

A quick decoding will reveal the actual code, which is a WSO shell, as I pasted in pastebin here-->[decoded]Dealing with WSO hack case, three points that you should know for reporting the incident are, the version, password and how you can successfully access it by understanding the condition needed to access this web-shell "hack" tool. All are summarized as per illustration below:Understanding these points will make you and the handler guy to know what step to do next.In additional, the below illustration are the most function that WSO being used for scanning for vulnerable service:

The other file, which was actually was last injected, is the rss-info.php an obfuscated PHP script with the snapshot below:This script was beautified with the note in here -->[pastebin], and decoded by our friends (Denis of SucuriLabs and @jseidl in--> [here] and [here] which is an attack script with acessing URL: counterstar .ws / c / counter .php which is a rotator to perform the active redirection (TDS) to some un-welcomed sites like one of the snapshot below:The rotator script is currently active and is worth to takedown. Additionally, I really do took time to silently monitor this rotator to get the any chance for the malware redirection or infection through it, but somehow the result still negative for an infection, they're lucky.

So now we know how they scan for directory and using the WSO for the remote access, using some malicious purpose on implementing redirection to a rotator site. Up to this point we can assume that the attacker can upload any script files with code to be remotely executed under the WordPress privileges, and that was the timing where the attacker uploaded the function_php.php and tempstyle.php, which are the point of this blog.

LD_PRELOAD - An introduction & PoC

I assume if you know and mostly you are using Linux, yes? Then you are familiar with the command of LD_PRELOAD, no? Below is the explanation about LD_PRELOAD for they who want to know:

One could override the C standard library's functions used in Linux (such as printf, fopen, open, rand, etc) with your own version of these functions, with or without calling the real library's functions after executing the overrided function.

Since the attack for this vector is already in the wild since March 2014 and many got hit by this threat too, I think it is important to disclose the fact of the threat for mitigating further. For the easy example I roughly typed the PoC in C code for "faking" or "infecting" an original UNIX fopen() command that can be "override" with one's coded program, but also to "intercept" it by executing a malicious codes (or program) before calling the original fopen():It's not not a new stuff & most of unixmen shold know this fact.That "#define _GNU_SORCE" directive instructs compiler to enable non-standard calls, in this case I need the RTLD_NEXT (at dlfcn.h) to be enabled to call the original version of "fopen()" calls.

If the code above is saved and compiled as a shared library like below command

gcc -Wall -fPIC -shared -o evilfopen.so evilfopen.c -ldl

The result of the C code above is we compiled a shared library (evilfopen.so) that can be used to intercept the fopen() with our own (fakes) code added with some malicious actions and then forcing the execution back to the original fopen(). The "evilfopen.so" can't be self-executed and need to be loaded to the provided enviromental API to intercept and to override, and the command for that is LD_PRELOAD. So, with the simple command line like below:

export LD_PRELOAD=$PATH/evilfopen.so

..the LD_PRELOAD command above will trigger "evilfopen.so" to intercept the fopen() (or other any function..depends on the attacker needs) to perform the malicious act, with or without executing the real function afterward. And this is actually exactly what had happened in this case, a malicious shared library dynamic ".SO" ELF shared library was dropped, installed & "LD_PRELOAD executed" to the targeted system to perform malicious action without the trace of its installation, which it may causing some headache.

What happened is: The LD_PRELOAD forces a defined libraries to be loaded for a program (library in our case) even if the program/binary itself does not ask for it. FYI, it is not a security flaw, since this scheme will work under one condition that if and only if ruid == euid (Real and Effective user ID), or, it is secured by the fact that the loader (ld.so) will ignore LD_PRELOAD if ruid != euid . In English: the only possibility to execute this attack is the gaining access of the shell with the existing user ID during the execution.

The installation of LD_PRELOAD "libworker.so" malware spotted

The latest version installer

The installation roles are executed by the files function_php.php and tempstyle.php which are identically coded with the malicious code as per below full code snapshot:

In the line 18 and 19 there are two variables contains big blob of the hexbin strings like below which is actually malware files, so I took a liberty to cut it for the explanation purpose:The explanation is as per follows:

Preparing the file_put_contents function to be used to inject hexbins..

This command will kill the host binary if it runs..

Hexbin in text strings for the 32bit ($so32) and 64bit ($so64) malware library SO files..

Checking whether the system is 32bit or 64bit architecture and assigning the $so variables to the blob hexbins string mentioned above (noted, the default is x64)..

How this malware script detecting the OS with reading the byte structure of /usr/bin/host binary:

And injecting the picked up hexbins to the file ./libworker.soIf the file extracted well, in my case it will show below output:

STDOUT;SO dumped 26848 .... for x32SO dumped 27288 .... for x64

The malware script detecting whether Mayhem (my favorite, kudos!) tools installed, if so it will quit here.. (read: chicken out), noted: be sure to run the tool under diferent environment settings :-)

Now, it's assigning remote requested URI to this evil script file($AU),initiating of the "host" binary name ($HBN) and current path ($SCP)

This is the very important part, the assembly of evil shell script file "1.sh" for the initiation of the evil library files dropped in previous process:

The self-generated script is below, with my comments for explanation:

The below codes are to make the script above to be injected into 1.sh file and make it full access with the world's executable privilege:

The below part of codes are explaining how the 1.sh is executed. It uses three ways to execute, by "at now" and "crontab" (with deleting its execution traces in at or crontab afterward) and additionally using common direct execution using "@system" PHP command, Noted: the comment was made by malware coder, not me.You will see how persistent the attacker in making sure the installation gone well..

Additional: The older version installer of malware shared library (libworker.so)

I was reported the older version installation script of the same attack series of libworker.so with LD_CONFIG, by using the different PHP installer, without using any 1.sh executable shell script. The overall "older" installer script itself can be viewed as per snapshot below:

The difference between the older version to the recent one is, it has many /* DEBUG */ comments which was written by the bad actor, and it has the additional that is having more functions, like autorun and FreeBSD ldd patch. The rest of the functions are similar to the recent codes in the "latest installer" section above. Below, I will explain these two differences with the arranged snippet code to make it more understandable:

Firstly, for FreeBSD system it added the "$extra" command parameter before executing the libworker.so in LD_PRELOAD. This can be assumed that the attacker want to make sure that the ELF shared library can be executed even in the FreeBSD servers.

The second difference is, the older version makes "autorun" scheme by injecting a command to itself in the rc.local, that can be implemented in both Linux and FreeBSD system. The snippet codes below is self explanatory:

Binary extracting sample scripts for IR & research purpose

For the Incident Handing folks of this threat, you can utilized the malware installer script to get as many .SO sample to extract to be submitted to Virus Total, below is the lazy script I assemble in PHP that fit to the purpose:

↑Please put the hexbin blobs from the $so32 or $so64 variables in malware installer PHP script into above template's $so*/$mo*/$no* accordingly and run it to get the extracted malware library sample files. The shell output in my case is as follows:

↑So few data, and my ELF reader meets another error too during analysis:

Error: Unable to read in 0x28 bytes of section headers.

Don't worry about this since the "malform" of the stripped binary itself is causing these annoying error messages. The malcoder (I can't stand it.. read:moronz) surely know how to strip clean a Linux binary.

Some notes in reversing: The binary is obfuscated divided in a row of dd calls like in below address (or section..or etc the name is..):

And this is the Segment type: "Externs" contains the NIX's C system calls to be used for the malicious purpose. Too bad they won't (maybe just can't..) encrypt it. The data is as per below list. It is important to write these all for the reference of behavior analysis part:

Later on proven by the behavior analysis, but the execution of this malicious library SO file will perform the backdoor operation, which forming the local information as contents to be sent to remote host using the template with some possibility contents, the initial callback (flagged as "R") and re-send (flagged as "P") has different contents, so does the fail confition (flagged as "F"). Additionally: researchers who in daily basis deal with botnets may understand this flagging scheme as the standard callbacks to a CNC or gates, and that knowhow help me a lot in reversing it:

PS: For the complete reversed libworker.so in disassembler format (using IDA Free with thank you to HexRays!) access here-->[MediaFire]

There are more on the deobfuscated assembly code like deletion routines, call itself with the external LD_PRELOAD (oh..not again?! why?). Networking operations, and so on.. The more details in the next section will describe the reason for all of those operation and proving the callback operation explained in above writing too..

Behaviour Analysis

UPDATES! The Video Tutorial of the behavior analysis is released here -->>[link]

Since the subject is the shared object (DYN) library, is a bit different to analyze than a static/dynamic executables or modular binary loaded by an application like web/ssh/etc server or etc, but this is the beauty of UNIX/Linux OS, all are transparent and only by using the standard UNIX debug environment anyone will be more than enough to recognize and stop this infection processes, which I will simulate in steps in this section.

For the details, after some tests I can summarize the execution of the evil binary is as per below:

- self-executing LD_PRELOAD on himself upon matched NIX function defined- forking (clone) some child processes- self-deletion after executing the POST data- dump the core to the memory and disk (I don't know why),in the disk after execution there will be .sd0 file saved.- leaving the trace in list of files (lsof)- leaving the LD_PRELOAD unset after infection (you'll see some LD_PRELOAD ERROR on libworker.so in "env" or "lsof" or "ps" since the "buggy" of the DYN libs malware for not including call to the "unset" code in the exit routine. :-)- sending callback contains user & system sensitive information to specific mothership URL via POST HTTP/1.0

The loading method is as per follows (as per script installer does):

$ export LD_PRELOAD=./libworker.so

You can check whether this libs is loaded by:

$ env[...]LD_PRELOAD=./libworker.so

What happened during running the /usr/bin/host?? It is good to know whether this evil libs is really loaded during execution. The below code is the debug mode of the loading process of libworker.so executed during the execution of /usr/bin/host. You can see that libworker.so was loaded in read only mode. The bits and size of the evil library, size=27,288 is showing it too. So it is being called preloaded.

I run lsof & grep rapidly monitorng the libworker.so, they look like also affecting trigger of malicious libs, so "by accident" it was forked as per below.. the size and memory allocated flag is a PoC of loading :-)

Breaking the myth (smile), below I lsof'ed the self forked process (32517) of this evil libs triggered by the "grep" command. I have my own reason since "grep" is using not so much shared libs during execution, it is good for the comparison & seeking of malicious calls or libs used by the evil library. Please see it carefully, I am in purpose grouping the output data:

Below is the original "uninfected" grep command itself in lsof, please seek for the difference the output below with the lsof result above, and you will see that libresolv-2.13.so, libnss_dns-2.13.so and libnss_files-2.13.so are loaded for the malicious purpose, which I don't think is needed in "grep" o.O? And so does the above's pts terminal socket which was replaced by the malware from "/dev/pts/x" into "/dev/null".. doh..I bet those malcoders think this can hide the mess they made from NIX admin's eyes..

This is the trace of the DELETION (executed by the unlink command in the malcode) already executed against the malware library file itself (libworker.so) in lsof. At "this" point the self-deletion was occured, to check it again , see the DEL in "type" column..

Yes, the libs was deleted, but the LD_CONFIG only being unset one time in the installation script, nevertheless the libworker.so was self-executing the fork of itself by LD_CONFIG'ing which explains the above errors.

The rest of malicious operation performed:

After being loaded as per traced in debug code above. Malware library executed itself after the original binary runs, I patched the debug trace during it runs with my comments as per follows:

There you go :-) I think you'll get a bit more idea by the above explanation about the malicious activities conducted. So let's see the PoC of those reversing and behavior analysis in its sent traffic :-). Let's move forward to networking parts!

Network Analysis

Either by reversing or network analysis, you will get the exact same values from one sample. Since this malware was hard-coded to have only static value per binary. For the best of all, let's capture the traffic gained by one malware, let's make sure whether this mess is really making a callback or not..Then I'll use one other sample to compare the result.

The first call, like we always face in the botnet cases, is very important and many initiation can be made, so I was very careful not to online the sample in the new environment until everything settled up. Everything? I mean you have the tcpdump, and you're good to go. You can capture this outside or inside the box (none of my analysis ever using inside the box traffic recording except for some testing though, my highly recommended: always discipline yourself to trap outside of the box)

..and this is the first call looks like:

Yes I test-installed this malware in some systems Linux and BSD in various versions, I have a lot of traffic like:

Only initial calls will get the flag R during the sent of the data, and the rest will be P flag like this:

Sometimes the malware itself got deleted during the operation by other fork.. so you'll see some "incomplete" traffic like this:

By testing the different sample I got the different traffic that goes to the same remote host in the different URL as per below:As per expected, each dropped SO library file is hard coded the specific CNC callback url :-)

I've mentioned in the installer part, there was an older version. That version was making a different CNC callback that can be shown in the below PCAP snapshot: The url is aiming a rotator script implemented in another hacked sites that can be used for many evil purposes.., and for your information the rotator is till UP.

In the infection that (still) occured recently (May 27th, 2014) we saw the attacker came from the OVH, France IP as per described in details in here-->[link]. Below is the callback image snapshot:

How to spot and stop the on-going infection?

I think I missed this part, so I added this now:

This malware, during the infection will make the dump data saved in the hard drive on the installed/executed directory with the file name of .sd0 , you can find with grep ".sd0" to your CMS directories to check whether your system is infected or not. There are cases where the hacked WordPress directory dropped by the malware .SO library (libworker.so) and not being executed, in this case there will be no ".sd0" file, instead you may can find the LD_PRELOAD installer script file of "1.sh" in compromised path. So searching for "1.sh" in the assumed compromised WordPress or etc CMS environment is good deed. Additionally, you can search for the "libworker.so" or "1.sh" by Google (Dork) since they are all accessible. Please see the video posted in the top of this post for the demonstration PoC for searching/confirming an infection from the remote using by a text editor and browser.

Below is the snapshot PoC of the searchable infection via Google search:

You can seek using (or make a signatre for this ELF malware .SO library) by these strings:

To dissect this running malware in a compromised server, you must find the installer script first, to find which library filename it will be executed. The name of the library will be made as lame as possible to fool us. Like in this case is as per below pic: If you find the installer PHP file, please remove it right away since that can be executed from the remote in anytime and will disturb the further cleaning processes.Together you can "secure" the other garbage that was being injected.

If you know the malware library name form the installer PHP file/script (i.e. libworker.so), then the rest is easier, just execute the below command:

$ lsof | grep 'libworker'

↑this should leaving you the PID of the processes that are utilized to execute the malware DYN library. You can kill (kill -9) that process' PID after beforehand making sure there are no vital services affected, to kill its "main malicious modules. You can do this after (or before) you check the enviromental setting in your box (env), and see whether the LD_PRELOAD is there or not, with or without error IF LD_PRELOAD is there execute the below command:

$ unset LD_PRELOAD

- right away, and repeat it again if you need to make sure no LD_PRELOAD exists in environmental variable anymore. Yes, by unsetting LD_PRELOAD and killing the affected PID will stop the overall malicious process respectively, so please DO NOT ever think to reboot the system with hope cleaning things up. :-))

A note for .sd0 "drops" file

Some people will think that the .sd0 file is malware drops or component of the malware, in my observation it is not it, is the garbage of malware dumped data, likely is in the coredump forms. When the malware .SO library is running, the .sd0 dump will fill the memory and the hard drive, it some systems I found the CPU resource was used because of this.

In some systems I investigated I found some .sd0 dropped in the directory where the malware .SO library was executed. In some systems I found some multiple execution of the .SO library malware files so I found more than one .sd0 files too. These .sd0 file' size is getting bigger each days, the biggest one I found was 16GB in size :-), so you don't want these files, if you find it please delete it. The good thing is this malware library (at the current version) always drops .sd0 files, so find this file if you want to be sure your system is infected or not infected.

Bad actor's CNC server:

All of the traffic lead to the IP address of 37.59.5.67 with sending POST command as per above PCAP information to the URL of:

The status of the both CNC URL as per today (Tue, 13 May 2014 08:23:06 GMT) i as per below snapshot:

The IP address of France IS: OVH: 37.59.5.67 is still serving CNC of this malware until NOW.

The URL http:// hennersy .com/sears/sears .php is down, good work!

The URL http:// hennersy .com/the-henner/hennesystuff .php is still UP and alive :-(

We urge to "related" law enforcement together to OVH to investigate this matter properly.

The disclosure of attacker IP Information

The ELF .so malware library's attacker IP address is easy to be identified after we know what pages in the WordPress that actually had been injected with malware code, or contains the WSO shell code inside. We are recording the attacker IP addresses from 15th February 2014 until today 13th February 2014, and with permission, we "masked" disclosing the attacker information in this section, as the next paragraphs stated in details. The point of this section is to pledge the help from the abuse team and law enforcement related to conduct the proper investigation and suspend the services that can be related to this attack. Full log of each attack IP is saved and can be used as evidence by the request from law enforcement agency only by the policy of the related entity.

For the wordpress access log, for the research and evidence purpose, I share the latest attack data detected as per today 13th May 2014 coming from Leaseweb Netherland (37.48.81.37, lw930.ua-hosting.com.ua) and Amazon AES, United States (23.23.222.104 , ec2-23-23-222-104.compute-1.amazonaws.com) as per below credential's masked records of two servers logs. // To be noted: // neostyle.php = malware installer PHP script// maink.php = WSO Shell version 2.5.1

Attack also coming from FTP vector

Today we received the evidence from the very different service that is infected with the same malware in their web service. This time the attack was spotted coming from the compromised FTP account. The libworker.so malware PHP installer script and the WSO version 2,5,1 PHP obfuscated script was spotted uploaded to a compromised site from the IP address IP: 5.39.222.141 (HOSTKEY.RU, Netherlands IP, NL-HOSTKEY-20120516) as per below report:This fact of evidence explains that only site with the Wordpress flaws aimed by the attacker, but the FTP leaked or hacked credentials are also aimed.

The moral of the story

This post is indeed an exhausting analysis writing, spent almost 3h testing, coding/decoding, with the rest of time for just writing it down..wow..English is hard, I even am trying to remember to make the title goes in right letter as per Martijn Grooten taught, but, most of all, I am so happy to do it :-). For the sake of the fellow UNIX administrator & security folks to get a good clue of what had hit them, and we can all think together on the best way to mitigate this. So I hope this post helps. But if somehow you find it not useful, don't mock but please try to advise on how we can improve with the idea or pointers, by writing the comment in the below of this post, which will be welcomed.

I really hope that the friends in Emerging Threat reading this to help in releasing the block rules for the traffic sent to the CNC since the header information are "very" unique and easy to block (will mail you later after cleaning some mess here, ok?). Additional: Thank's for releasing the rule, friends!

Please tune your system, don't leave it un-attended, if you use UNIX service please learn what the "do's" and "don'ts" of the services, and for Christ sake please NEVER open any directory with permission of #777, ok? Contact WordPress folks properly for some guidance, help and support if you have problem with the uploads directory with default permission.

If you got hit by the files described in illustration at the top of this post, your system is practically compromised, if you find the injected files in WordPress directory working area means your CMS is compromised. If you find WSO or other nasty things like C99 shell installed in your CMS area, the damage can be worse, you should audit the system right away. You can not think is secure to continue the service anymore unless you clean up the injected files, killing malicious process, fix the security holes, run security audit to make sure everything is in a proper way. So please make announce, then store backup system up to replace the compromised one and get the compromised system offline for the clean up purpose. I knew some systems that has to be rebuilt for having a terrible damage too :-(

To be sure to know WHY the attacker can get in is very important to fix the security holes, and for that, read all related logs is the best! Also search internet for the stuff you find in the logs or injected files, it helps you to clean up the mess. After everything clean, you can run it again. If you don't fix the security holes now, you are not just opening the risk for the attacker or next attackers will come to visit you again and may bring more damage to you, but you are ALSO risking your server to be utilized to damage others too via malicious activities like unwanted traffic redirection, exploitation, and malware infection.So please conduct as per described above. If you have any question, leave it in the comment please, myself of other from our team may reply you properly.

Virus Total detection and Samples

Friends, please do not hope or count much on the UNIX/Linux AntiVirus Scanner for scanning the threat's files since the detection ratio of this binary series is very low (as per stated in following samples/lines) and the way I observed after seeing nearly 40 units of infection, I think in every campaign it has different ".SO" binaries and injected PHP installers. Since the CNC's URL is different in some binaries..which "currently" lead us to the hacked server used as CNC, we can assume that the bad guys is now re-compiling more and more malware library to avoid signature detection by just changing the URL string and hit one gcc command line, hence a new low or FUD (Full UnDetected) malware library will be released easily.. :-(

Therefore, I wrote this long post. Understanding the how it works is very important. By this information a professional NIX admins you can build a scheme for mitigation and prevention method to secure their system. And you can write about that know-how in your blog to share the knowledge to help others too. :-)

To loyal friends who keep on motivating MMD spirit in research, thank's & won't let you down! :-)@wirehack7 for renting testing environment since my systems are all BSD basis only.@jseidl & Denis (of UnmaskParasites or SucuriLabs) for helping decoding accompanied obfs. redirector. PS: Share the script friend!To fellow admins in IDC & Mr. Shohei Maeda for allowing me to disclose this malware fact to the world.To the open source community & wonderful *NIX/Linux people who provide good reading for research. And to my family for patiently letting (or ignoring) me by sitting almost 12h for writing this..