WordPress, and the Pingback of Death

The journey to discover why I couldn't keep a website up.

I host a number of websites for clients, friends, and family. A solid number of those are running WordPress.

I rarely suffer problems with them... except for one site. This site has been going down, and staying down, to the point that I routinely SSH in to forceably restart the locked-up PHP processes.

I've tried to fix it in the past to not avail. Previously I've migrated the site to a new host with an updated OS, tweakes a great many configuration settings all over the system and site, and very recently I've changed the server setup to match that of other high-traffic WordPress sites I host.

But the lockups have been increasing with frequency to the point where, today, the site would not stay up for more than 30 minutes before refusing connections.

Finally fed up, this is my journey to fix it.

I've been fighting this for quite a while with no traditional remedy in sight, so I immediately dragged out the big guns, and strace one of the locked processes (which prints out every system call a process makes):

This is a pingback; http://www.remote.net is kindly letting us know that it has linked to http://example.com/somepage. As part of the protocol, we then open that URL and look for the link to verify that this pingback is accurate.

We open an HTTP request to the URL, and get back a block of obfuscated JavaScript. Then, the connection blocks forever.

My best guess is that there is a bug in WordPress' HTTP request model in this version. If the connection doesn't close, then WordPress hangs on it. When the remote server doesn't close the connection due to either (1) a bug, or (2) on purpose, we wait forever.

I can't tell which of those it is, but I can de-obfuscate that JavaScript (and if you swap out the eval for console.log (and run the code), you get window.location=qi+"/?jdfwkey=1kjv92"+ri). I have no idea what this is supposed to do.

At this point I have "solved" the problem by disabling XML-RPC on this WordPress install. I would love to keep poking at it to come to a fuller picture, but I'd rather keep this site up than satisfy my absolute curiosity.

(Hostnames and IPs have been changed to mask the identity of remote hosts, as I do not currently know where their allegiances lie.)