Email a friend

To

From

Thank you

Sorry

It's one thing to write about hackers, scammers, and malware, as I've done for various venues for the last 15 years. It's quite another to experience the nastiness first hand. Yesterday it was my turn. Here's my story.

Last night around 7 pm I Iogged onto my occasionally NSFW humor site, eSarcasm, to post something snarky. What I saw though, was something different and much scarier than usual -- A big fat warning sign from Google Chrome:

We tried other browsers. Firefox displayed a magenta page that directed users to StopBadware.com, a site co-sponsored by Google and Mozilla designed to steer people away from malicious Web sites. My Firefox NoScript plug in confirmed that, sure enough, a script trying to redirect to brnighome.com was running on our home page, but only some of the time. And we could find no evidence of any links to brnighome.com anywhere on the site.

Fortunately Frank F., our man at Doreo, was on the job. WIthin an hour he had tracked down the problem. Our OpenX software -- one of the programs we use to serve up rotating ads and banners on our site -- had been hacked. Two new admin accounts had been created, both with IP addresses based in Germany. And these two arschlochs had inserted a tiny bit of Javascript code into our banners, so that anyone who clicked on them would be redirected to a malicious site that might do god knows what to their computers -- most likely install malware.

Turns out this was not an uncommon problem with OpenX. For the past year attackers have routinely exploited vulnerabilities in OpenX to serve up 'malvertising' to unsuspecting users. To its credit, OpenX has patched these holes as quickly as it finds them. Unfortunately, we didn't know about these vulnerabilities, and we had not patched our software. We simply set up OpenX to rotate some banners and forgot about it. That was a mistake.

Getting rid of the malware was as easy as getting rid of OpenX. (Hasta la vista, baby. Don't let the virtual screen door hit you on the way out.)

A bigger problem? Getting our site off of Google's blacklist. Even after we'd gotten rid of the malicious code, visitors to our site were still seeing those scary red screens telling the world we were bad bad webmasters who must be shunned. And this was happening on a day that the Google gods had been smiling down upon us, sending us lots of traffic.

It's like getting that cute guy or girl down the hall to finally notice you, on the day you've got a big juicy canker sore on your lip.

In a word, oy. But it gets worse.

My personal site, dantynan.com, was also red flagged. Why? Because I'd installed a widget that served up a scrolling list of headlines from eSarcasm. This meant any other site that had installed that same widget (a few hundred at last count) would also display the Red Screen of Death. This was very bad.

Fortunately, StopBadware has a simple process for reviewing sites that have cleaned up the nasty bits from a hack attack. We submitted our site and waited.

Google's Webmaster tools dashboard is also supposed to offer an option for re-reviewing your site after it has detected nasty stuff on it. This option was invisible for us (and we are not the only ones). Fortunately I discovered a solution -- reloading the dashboard from scratch reloads the initial "Request review" button under the Diagnostics/Malware tab.

According to Google, it could take as long as 48 hours to get off the blacklist. That would have been a disaster for us. But we were lucky. Within 8 hours, the ban had been lifted.

Now all we had left to do was tell our users what happened, and hope that a) no one had gotten infected by visting our site, and b) people who'd encountered the Red Screen of Death would be willing to come back again. The jury's still out on both of those.

Lessons learned? Obviously keep all your site's plug ins and software up to date. (Though this would have been much easier if OpenX had plugged into the WordPress admin dashboard and not required its own, or if it had some mechanism for alerting users when security holes had been discovered besides its support forums.)

A more important lesson though, is that the price of Internet publishing is eternal vigilance. Being hacked can happen to anyone. Even you.