Monthly Archives: August 2011

Post navigation

My wife, Kerry, went down to California last thanksgiving for a brief trip with her sister Sandy. They were driving along in Fountain Valley with my niece and spotted a kitten lying in the gutter. As they approached her she hauled herself out of the gutter in a badly injured way.

They immediately stopped and tried to get the kitten out from the bush she’d crawled into. The neighbors said she’d been hiding in that bush for at least 2 days. Animal control had been out and couldn’t catch her.

It took the three of them 45 minutes to get her out with my wife crawling under the bush coaxing her the whole time. She eventually caught the little thing and they took her to the Vet ER.

Turns out Stix, as she came to be called, had a broken pelvis. She was 9 weeks old. Three breaks, with the largest one highlighted on the xray below. This xray links to the hi-res full body xray if you’re interested.

Kerry and Sandy had to leave immediately to drive back to Colorado from California and they took Stix with in a cat carrier.

Stix was paralyzed from the base of her tail down and Kerry was told the kitten may not be able to use the bathroom properly. On the drive back to Colorado, in a Vegas hotel room the first time Stix peed in the catbox there were cheers. Later that night she pooped, and more cheers. In that hotel room when Kerry cleaned her cat carrier the blanket was covered in dead fleas that had fallen off from the anti-flea medication the vet gave her. She also wolfed down her first meal like she was starving.

Kerry started by saying she’s just going to foster her and find her a good home once she’s recovered. The road to recovery was long. Kerry kept her in a cat carrier initially. As she healed she bought a large pet playpen and kept her in there to limit her movement. She recovered further and eventually got to roam around the house supervised.

Of course attachments grew and she became our cat.

The first time she lifted her paralyzed tail slightly was a big day. A few months later we were watching TV with her and she actually twitched the end of her tail and we were shocked. More cheers.

It’s now 9 months later and she’s the new favorite child of our three pets. We have an australian cattle dog and Stix doesn’t realize she’s a cat. She imitates our cattle dog so much we’ve called her a cattle-cat. When Joey chases the ball, she goes after him. And she follows us around when we take Joey for a walk. That’s in between wrestling with our oversized orange tabby.

This is a video of her at the end of a 20 minute walk in 90 degree heat out here in Colorado. She’s very camera shy so it’s taken me a while just to get this footage. I’ve never known a cat that comes when you call her. She’s awesome and for starting life with a broken pelvis she’s turned into a great hiker and runner which is highly unusual for a cat.

Our cattledog has always been partial to me – they tend to be a one person animal. When Kerry is sitting reading, Stix will jump on her chest and snuggle right under her chin. So she jokes she finally has a pet of her own.

Not all rescues have a happy ending like this and the prognosis for Stix was very bad when Kerry and Sandy rescued her. But I see similar stories online and I think animals are tougher than humans because they don’t know that they’re injured or disabled and so they lack self-pity. As far as Stix is concerned, everyone else’s tail is just a little too active and for some reason in the last few months her back legs got all grasshoppery and now she just wants to run everywhere.

I spent some time yesterday reaching out to folks I know to try and get some input on WordPress security, avoiding getting listed as Malware and how to get removed from the Malware list. Rand Fishkin, the founder of SEOMoz and all round SEO God was kind enough to introduce me to Justin Briggs who is an SEO consultant and guru. Justin quickly came back with the following advice:

WordPress is certainly more susceptible to malicious attacks due to its popularity and the large number of sites that can be compromised with an exploit.

The best preemptive solution is to keep up on updates and increase security associated with WordPress.

I’ve had a friend’s site who was exploited several months ago. It was a bit of work to get it cleaned up, but the warning was removed relatively quick after submitting the request to Google.

I contacted friends who are current and former Google employees but no luck getting in touch with the Malware team. In general it’s hard to connect with folks inside the big G with questions that are usually handled by support teams. [As I’ve been politely told in the past]. 🙂

I’ve had two reports in the last 12 hours of WordPress blogs that were compromised via the Timthumb hack being listed as malware by Google. If you try to visit either site, you are confronted with the following:

These sites are listed with the warning that “This site may harm your computer” in Google’s search results and Google blocks access to the site with a warning forcing you to manually type the URL into your location bar if you really do want to visit the site:

One of the site owners sent me the detailed info that Google Webmaster Tools was giving her:

This malicious code is appearing intermittently on this author’s WordPress site. I’ve seen this same pattern recently in blogs I’ve repaired and the way it works is that the site is periodically downloading new PHP code from a remote site run by the attacker and re-injecting it into the wordpress code. That allows the attacker to add, remove and update whatever code he/she is executing on your blog. So they could for example update any spam links every few hours.

To prevent your site being listed as malware clean it as fast as possible

The fastest way to do this, although it doesn’t gaurantee a complete clean, is the following:

Remove all old plugins and themes you aren’t using.

Upgrade all your plugins and themes to the latest versions and make sure none of them use an old version of Timthumb.

Clean any Timthumb cache directories.

Upgrade your entire wordpress installation, even if it’s at the latest version. This overwrites all wordpress files.

Search your directory tree for any remaining suspicious files that contain base64_decode wrapped in an eval() statement or URL encoded data. More info on how to do this search here. Delete any files you find. NOTE: If you don’t find any additional infected files in this step, it’s highly likely that your site is not clean. Every attack that I’ve seen so far using Timthumb gets in by uploading a file into the cache directory and then uploads an additional file into a writeable directory on the blog to ensure continued access once the cache is cleaned. Make sure you find that additional file.

Make sure the only directory that is writeable in your wordpress installation is wp-content/. Directories like wp-admin and wp-includes should be read only by the web server.

Potential long term impact of this vulnerability

The fact that I’ve seen the same domain being used by attackers on multiple blogs suggests this attack may be partially or fully automated. The worst case scenario is that we end up with a WordPress botnet with thousands or tens of thousands of servers on high bandwidth links compromised and able to send spam emails or launch a huge DDoS attack.

Keep in mind that most botnets are compromised windows machines on relatively slow home broadband connections. Their uplink speeds are around 512kbps. These WordPress servers are on links that are a minimum of 10 Megabits per second each, so they have plenty of firepower for a coordinated attack. One WordPress server is equal to at least 20 infected PC’s in terms of pure bandwidth firepower.

I was contacted by another site owner who was hacked via vulnerable WordPress themes today. He had updated to the latest non-vulnerable version of his theme, but the WordPress theme installation or update process doesn’t remove, or remind you to remove, old themes that may be vulnerable. So while they encourage you to update everything, old versions are still lurking on your site waiting for an attacker to take advantage of them.

Remember: Delete all old unused themes and plugins.

In this case an attacker once again used an old version of timthumb to install an attack shell called Sniper_SA. The attack shell was Arabic so I’m assuming the attack came from an Arabic country. [The last 3 I’ve seen were english]. This one was base64 encoded inside a PHP eval.

The web host is one of the top 3 WordPress.org hosts on the web. Their default installation is to have your entire WordPress installation writeable by the web server and the server can even write to your home directory under the web root. This opens up all sorts of possibilities for a hacker to gain a remote shell. WordPress hosts, please secure your default WordPress installations so that only directories under wp-content/ are writeable. Also make sure the user’s home directory is not writeable by the web server by default.

Exec Summary: Today I’m launching a Beta open source project called DeployMint. I’m using it on WordPress installations where WordPress is being used as a CMS. It runs as a WordPress plugin and allows for staging and deployment of WordPress sites along with robust version control and zero down-time during deployments. It uses the Git version control system to store site snapshots in a safe and space efficient way. It also takes a “belt and braces” approach and provides an emergency back-out system separate to Git in case a deployment fails. You can download the latest version of DeployMint and see a video demo at the DeployMint project page on Google Code.

Full blog entry:

My company is busy moving to using WordPress as a CMS and I wanted a way to instantly deploy several new pages of content or an entire site and have dev and staging sites to test new ideas. I also wanted version control, instant deployment and an emergency back-out system.

I also needed comments to be preserved on the live site so that if I deploy a new version, the existing comments on the live site stay where they should be and only page or post content is updated.

DeployMint runs under WordPress MU. You create as many subdomains as you would like, for example:

development.example.com

staging.example.com

example.com (your live site)

Then you design your entire site with themes, pages, content on development.example.com.

Once you’re done, you take a snapshot of development.example.com and deploy that snapshot to staging.example.com.

Your client reviews the new site on staging.example.com and suggests changes.

You make the changes on development.example.com, take a new snapshot and deploy that snapshot to staging.example.com

Once your client is happy, you take a snapshot of staging.example.com and deploy it to example.com, your live site.

Here is a video showing the basic functionality of DeployMint. DeployMint is installed on this blog and I use it to test out new themes and design changes. It works as well on a blog or when WordPress is being used as a CMS. In this video I take a snapshot of my live site (this blog) and deploy it to my staging site staging.markmaunder.com. Then I make a minor modification, I re-snapshot the staging blog and deploy that snapshot to the live site.

DeployMint is space efficient because it uses Git to store snapshots. It also makes a full copy of your entire WordPress database including all your WordPress MU sites every time you deploy. Because these require more space, you can choose how many of these full backups you want to keep. If things go awry with your database or deployment for some reason, you have an emergency backout system that will restore your WordPress MU installation to the state it was in before your previous deployment.

Behind the scenes, DeployMint (DM) works as follows:

To install DeployMint you need to create a data directory that is not under your web root, but is writable by your web server.

When you create a new project, a new Git repository is created.

When you take a snapshot, DM dumps all tables belonging to the blog you snapshotted into individual files.

Those files are checked into a ‘Git’ repository. DM uses git for storage because it’s space efficient and robust.

Every snapshot you create is a new branch in the repository and only the changes are stored.

When you deploy using DM, it simply checks out the branch you want to deploy and imports it into a temporary database.

In that temp database, we merge all existing comments on your site into the site we’re about to deploy.

DM also modifies any hostnames it needs to, to reflect the site we’re about to deploy to’s hostname.

Before deployment, DM takes a full backup of your entire WordPress MU database including all sites and stores this for emergencies in case you need to back-out your changes.

These backups take up more disk space than snapshots, so you can choose how many of them you want to keep and DM auto-deletes the oldest ones first.

Then a rename is done which takes a few hundredths of a second to replace your old database with the new database we’re deploying.

And you’re live with your new site!

Please post a comment below if you have any features suggestions or comments. Thanks.

I just helped another target of the timthumb.php vulnerability to clean their machine. The method the hacker used to hide their tracks was a little different to what I’ve seen in the past. So I wanted to mention it here and let you know how to scan for it.

As I previously mentioned, the method I’ve seen hackers use to hide their source code is to encode it using base64 encoding and then use base64_decode and eval() in PHP to execute the code at runtime.

You can scan for base64 decoding by getting a shell on your WordPress server and running the following in the root of the WordPress installation directory:

grep -r base64_decode *

Keep in mind that some files that are not hacked will show up, like the newest version of timthumb.php which includes a base64 encoded image. But this is a good starting point to get a list of files that warrant further inspection.

The hack I saw today was different. The hacker used hexadecimal escaping to hide their tracks. They didn’t just encode hostnames and things that a security analyst would obviously search for. They also encoded individual javascript commands and strings containing HTML element names.

You can use this perl compatible regular expression to search for hex encoded data in your javascript. Again, run this in a shell in the wordpress root installation directory:

grep -rP “(?:\\\\x[A-F0-9]{2}){5}” *

This will search for strings of at least 5 sequential hex encoded digits. You may get some false positives like class-simplepie.php . But again, this will give you a list of files that require closer inspection.

The file that was infected today was wp-includes/js/l10n.js. The attacker had appended hex encoded javascript to it. You can see what a normal file looks like here.

If you’ve been hacked, or suspect you’ve been hacked, drop me an email at mmaunder at gmail. I charge a very reasonable consulting rate and it usually takes 1 to 3 hours to fix the system and harden up permissions to prevent future attacks.

John Robb writes an excellent post arguing that the concentration of wealth in the United States has resulted in a centrally planned economy. I wanted to expand on his writing.

After World War II, there was a widely held view that Nazi Germany was the result of failed capitalism. Economists and political scientists in the UK and across much of western Europe thought that Capitalism was a bad thing and the answer was socialism.

A now famous economist called Fredreich von Hayek argued in The Road to Serfdom, published in the early 1940’s, that Nazi Germany was actually the result of central planning. He suggested that a centrally planned government is destined to become fragile and is easily seized and taken over by those that might not play by the rules.

Hayek was based in England, but his book was far more popular in the United States and it may be the reason we ended up with a free market economy post WWII.

John Robb’s idea is a new and useful lens to examine our political and economic decline: Through capitalism gone wild, we may have ended up with all the trappings of socialism after all.

This is the other point of view. An interview the BBC probably won’t air again.

I don’t condone violent demonstrations and I think the the looting of small businesses is sad and immoral. But you should understand that sometimes when people hit the streets en masse and make some noise, it has a purpose. It can’t be explained away by labeling them “rioters”.

It happened during apartheid in South African where I grew up and it brought about a peaceful transition of power in the South African government.

Back then we used to call Nelson Mandela a terrorist. Today he is Madiba, one of the most loved humans on Earth.

If you run a reasonably high traffic blog on a small Linode server like this one, it’s a really good idea to set up an Nginx front-end proxy for your Apache server. It lets you handle relatively high traffic without running out of apache children while keeping keep-alive enabled.