Updates on CyberSecurity, WordPress and what we're cooking in the lab today.

Hacking 27% of the Web via WordPress Auto-Update

At Wordfence, we continually look for security vulnerabilities in the third party plugins and themes that are widely used by the WordPress community. In addition to this research, we regularly examine WordPress core and the related wordpress.org systems. Recently we discovered a major vulnerability that could have caused a mass compromise of the majority of WordPress sites.

The vulnerability we describe below may have allowed an attacker to use the WordPress auto-update function, which is turned on by default, to deploy malware to up to 27% of the Web at once.

Choosing the most damaging target to attack

The server api.wordpress.org (or servers) has an important role in the WordPress ecosystem: it releases automatic updates for WordPress websites. Every WordPress installation makes a request to this server about once an hour to check for plugin, theme, or WordPress core updates. The response from this server contains information about any newer versions that may be available, including if the plugin, theme or core needs to be updated automatically. It also includes a URL to download and install the updated software.

Compromising this server could allow an attacker to supply their own URL to download and install software to WordPress websites, automatically. This provides a way for an attacker to mass-compromise WordPress websites through the auto-update mechanism supplied by api.wordpress.org. This is all possible because WordPress itself provides no signature verification of the software being installed. It will trust any URL and any package that is supplied by api.wordpress.org.

WordPress powers approximately 27% of all websites on the Internet. According to the WordPress documentation: “By default, every site has automatic updates enabled for minor core releases and translation files.”. By compromising api.wordpress.org, an attacker could conceivably compromise more than a quarter of the websites worldwide in one stroke.

Below we describe the technical details of a serious security vulnerability that we uncovered earlier this year that could compromise api.wordpress.org. We reported this vulnerability to the WordPress team via HackerOne. They fixed the vulnerability within a few hours of acknowledging the report. They have also awarded Wordfence lead developer Matt Barry a bounty for discovering and reporting it.

Technical Details of the api.wordpress.org vulnerability

api.wordpress.org has a GitHub webhook that allows WordPress core developers to sync their code to the wordpress.org SVN repository. This allows them to use GitHub as their source code repository. Then, when they commit a change to GitHub it will reach out and hit a URL on api.wordpress.org which then triggers a process on api.wordpress.org that brings down the latest code that was just added to GitHub.

The URL that GitHub contacts on api.wordpress.org is called a ‘webhook’ and is written in PHP. The PHP for this webhook is open source and can be found in this repository. We analyzed this code and found a vulnerability that could allow an attacker to execute their own code on api.wordpress.org and gain access to api.wordpress.org. This is called a remote code execution vulnerability or RCE.

When a request arrives at api.wordpress.org, presumably from GitHub, the api.wordpress.org webhook verifies that it is in fact GitHub making the request by using a shared secret and hashing algorithm. The way this works is that, as GitHub is about to send JSON data, it combines the data it’s about to send with a secret value that has been pre-shared with api.wordpress.org. It then hashes the combination and it sends that hash along with the JSON data to api.wordpress.org.

When api.wordpress.org receives the request, it takes the JSON data, also combines it with the shared secret and computes a hash of its own. If its hash matches the hash that GitHub just sent, then GitHub must know the shared secret and that proves that GitHub is allowed to make the request it’s making.

GitHub uses SHA1 to generate the hash and supplies the signature in a header: X-Hub-Signature: sha1={hash}. The webhook extracts both the algorithm, in this case ‘sha1’, and the hash to verify the signature. The vulnerability here lies in the fact the code will use the hash function supplied by the client, normally github. That means that, whether it’s GitHub or an attacker hitting the webhook, they get to specify which hashing algorithm is used to verify the message authenticity, as you can see in the code below.

If we can bypass the webhook authentication mechanism, there is a POST parameter for the GitHub project URL that is passed unescaped to shell_exec which allows us to execute shell commands on api.wordpress.org. This allows us to compromise the server. You can see the shell_exec call in the code sample below:

The challenge here is to somehow fool the webhook into thinking that we know the shared secret that GitHub knows. That means that we need to send a hash with our message that ‘checks out’. In other words it appears to be a hash of the message we’re sending and the secret value that only api.wordpress.org and GitHub know – the shared secret.

As we pointed out above, the webhook lets us choose our own hashing algorithm. PHP provides a number of non-cryptographically secure hashing functions like crc32, fnv32 and adler32, which generate a 32bit hash vs the expected 160 bit hash generated by SHA1. These hashing functions are checksums which are designed to catch data transmission errors and be highly performant with large inputs. They are not designed to provide security.

If we can find a weak enough hashing algorithm, it lets us brute force attack the webhook. We just need to send a series of hashes, trying to guess the hash value of the shared secret and the data we’re sending, until we get it right and api.wordpress.org allows the request.

By using any one of these weak hashing algorithms, we reduce the brute force space, or number of guesses we have to make, significantly (2^160 down to 2^32). But it is still not feasible to launch this attack over the network to api.wordpress.org without it being incredibly noisy as we make a large number of guesses.

Of these weak algorithms, the one that stood out the most was adler32, which is actually two 16 bit hashing functions with their outputs concatenated together. It also has a known implementation flaw when used on short messages. When combined with PHP’s hash_hmac function, the second round of hashing will only ever pass 68 bytes of data to adler32 which severely limits the amount of possible hashes that can be generated given any input at the start of the hashing process.

Not only are the total number of hashes limited, but there’s also significant non-uniformity in the hash space. This results in many hashes being the same even though they were supplied with different inputs. The distribution of possible checksum values are similar to rolling dice where 7 is the most likely outcome (the median value), and the probability of rolling any value in that range would work its way out from the median value (6 and 8 would have the next highest probability, and on it goes to 2 and 12).

The proof of concept supplied in the report utilizes the non-uniformity by creating a profile of most common significant bytes in each 16 bit hash generated. Using this, we were able to reduce the amount of requests from 2^32 to approximately 100,000 to 400,000 based on our tests with randomly generated keys.

This is a far more manageable number of guesses that we would need to send to the webhook on api.wordpress.org which could be made over the course of a few hours. Once the webhook allows the request, the attack executes a shell command on api.wordpress.org which gives us access to the underlying operating system and api.wordpress.org is compromised.

From there an attacker could conceivably create their own update for all WordPress websites and distribute a backdoor and other malicious code to more than one quarter of the Web. They would also be able to disable subsequent auto-updates so that the WordPress team would lose the ability to deploy a fix to affected websites.

Are You at Risk?

We confidentially reported this vulnerability on September 2nd to Automattic and they pushed a fix to the code repository on September 7th. Presumably the same fix had been deployed to production before then.

We still consider api.wordpress.org a single point of failure when distributing WordPress core, plugins and theme updates. We have made attempts to start a conversation with members of Automattic’s security team about improving the security posture of the automatic update system, but we have not yet received a response.

Should api.wordpress.org or a similarly critical server be compromised, it is possible that an attacker could compromise WordPress sites en-masse across the web in a very short amount of time and simultaneously remove the ability for the WordPress/Automattic security team to push out a security update or a fix.

This would clearly be a disaster for the Web and the online community. The phrase “too big to fail” comes to mind. With 27% market share, WordPress powers over a quarter of all websites. A failure of this magnitude would be catastrophic for the Web. Furthermore, it would provide a massive attack platform for the attacker, who would control millions of web hosting accounts from which they could launch further attacks.

Credits

We would like to congratulate Lead Wordfence Developer Matt Barry for this spectacular research. Matt tackled this project in his spare time and handled the disclosure of this via HackerOne to the Automattic team. Matt is also the technical author of this post with some minor assistance from the rest of the Wordfence team.

"We have made attempts to start a conversation with members of Automattic’s security team about improving the security posture of the automatic update system, but we have not yet received a response."

That pretty much sums up the attitude and arrogance of Supreme Emperor Matt and his entire team at Automattic (and .org core as well). They all know best and the hell with anyone else and their pesky "advice".

As much as I'd hate to be on the receiving end of this type of web "correction", having 27% of the web compromised would sure be an attitude adjuster for Supreme Emperor Matt (and the instant death of WordPress and Automattic unfortunately). That's a lot of eggs in one basket.

Clearly this issue is only patched up for now. Looks like there are still some pretty major concerns. Good work WordFence!

I didn't "fail to mention" anything. It was mentioned in the article. Did YOU miss it?

Did YOU miss the fact that it took 4 DAYS for them to even acknowledge it?

This has been discussed for 3 YEARS yet they failed to act until there was a clear threat. They're just bullshit lucky that WordFence found it rather than one of a million black hat hackers out there. We wouldn't be having this conversation in that case - we'd all be trying to fix our stuff (or looking for an alternative to WP).

Well, there is no known ongoing vulnerability. But one can be discovered by some bad guys, soon, you never know. The point is: it is absolutely necessary to improve the security posture of the automatic update system as soon as possible. It should have been done years ago.

Would you suggest that we disable the automatic updates feature entirely, until such time as a more secure system is deployed by WordPress? The article above states:

"We still consider api.wordpress.org a single point of failure when distributing WordPress core, plugins and theme updates. We have made attempts to start a conversation with members of Automattic’s security team about improving the security posture of the automatic update system, but we have not yet received a response."

That seems to imply that it may be advantageous for those of us who have only a few WP sites to manage (8 in my case) to return to doing manual updates.

Alternately, we could manually trigger plugin updates from Wordpress.com (which I do anyway, when they show as not having already been updated). All of my manages sites (6 of which are mine, 2 of another organization's) are hosted by GoDaddy. They offer update capability (automatic or manual) from the C-Panel.

1 - Are there any known vulnerabilities in using these methods to update things?

2 - Would these methods bypass some of the inherent weaknesses of the current WP update system entirely?

I would appreciate it if you would expand the points of the article (particularly the part that I quoted above) by recommending more secure approaches than simply keeping WP's automatic updates active. Thanks.

Thank you guys and gals at WordFence ALSO for all that you do with respect to not only the plugin, but your concern about the security of WordPress sites everywhere! You all are truly community-minded, and many of us appreciate that tremendously. :)

Superb article and excellent work uncovering this vulnerability. Should site owners who have set WordPress core to automatically update on their site, now consider changing this? It would be very helpful to have some commentary on this from the WordFence team.

I'm not referring to the automatic CHECKING for updates (which is built into wordpress) but rather the updating process itself - automatic vs requiring approval.

Some people have their WordPress installation set to update the core automatically (to implement security fixes as quickly as possible on their sites), but from what has been said in the article, it's now clear that this is a security risk because in the event that the api is compromised, malware would be uploaded equally quickly. Comments?

Despite the risks that are accompanied with hamfisted attempts to build an automatic update feature, they are unambiguously a good thing. Enormous security win. But security has to be part of the design, not an afterthought.

Even if anyone disregards the advice given and decides to disable WordPress's automatic update feature, do not demonize automatic updates in general. There are ways to do it securely.

We're not recommending you do that at this time. Already replied to another commenter about this. This is really a double edged sword:

If you leave auto-update enabled, then there's a very small chance that WP infrastructure is compromised and something bad comes down the pipeline. This is a very very small risk and the issue we described above has been fixed by WP. The benefit though is that if you have a new severe vulnerability in WP core or a theme or plugin, then you will benefit from an auto-update fix which will be pushed out by WP.

On the flip side, if you disable auto-update, you're removing the risk of a malicious update, but it's important you understand that this risk is very very small. But you are now also not benefiting from auto-updates of vulnerable code. Because it's much more likely that a severe vulnerability in core or a popular plugin or theme will be found that requires an auto-update, we recommend you leave auto-update enabled for now.

Thanks so much for your great work on this potential problem. I had the same question about disabling auto update. I think it would be good for you to put your explanation of your recommendation at the end of the article instead of at the end of the comments. Not everyone reads the comments. Thanks

Auto-update bring another problem. Any new line of code could be a problem without a test in a stage environment. Put it in production just because "the creators" wrote the code is. IMHO, be naive. They can win 10, 15, 20 times but, if one single day they lost, everybody goes down together.

Bloggers and small companies, for sure, don't have this kind of "care policy" about updates but, in large environments and installations this is something common and there, nobody use automatic updates (I speaking over my knowledge).

Personally I don't like automatic stuff. Instead that, I prefer to use some tools to check what's happening and, if I need an update, make it in a stage environment before production. Shit always happens.

You guys continue to show your expert handling of web security matters! You provide a continual demonstration of high expertise and the discovering of potential security problems. You guys rock the world!

This was an obvious problem when the automatic updates first began. It will not go away. There will always be something in there for an exploit, waiting to be discovered. The stakes are high. You can bet there are thousands of smart criminals trying to find a way to exploit the automatic Wordpress updates. They are a terrible idea.

So glad to hear that you confidentially communicated the problem to Wordpress and let them fix it before you published it. Years ago, there were these 3 hacker-brothers-turned-penetration-testers. One of them kept seeing an ad for a shopping cart program, one that I happened to be using on my ecom site. He decided to see if he could hack it, and found a backdoor that the dev company used to reset lost passwords for their customers.

They posted the info on their website, a site that many hackers read, looking for just such information. And the chaos began. They notified the dev company, but not before hundreds of carts were hacked. One common thing the hackers would do was to add their own email address to the list of addresses to receive orders. Back then, things weren’t quite as anonymous as they are now, and I received several emails from people saying things like “please don’t prosecute me, it was an initiation I had to do” (to get their hacker wings, I guess). Or warnings from friendly tech-savvy visitors, telling me that my cart was insecure.

So kudos to you for letting them fix the problem before announcing it!

Wordpress has been around for a while now (and I'd imagine using this simplistic update scheme) and big in the news in recent years for securing so many websites; it's incredible to me that this hasn't already been exploited; or, is it that the exploit is there, just waiting "in the shadows" for someone to pull the trigger?

I'd imagine that someone (or group) would have already found this as it seems so openly obvious... at least game theory / Art of War would have you believe it's obvious.

Anyway, kudos to Matt / WordFence for finding it (& possibly forcing WordPress' hand to revisit their update model by releasing this public display of vulnerability :))

Well Mark your team has done it again. In spades this time.
Sadly, due to losing my job I had to let my paid license go for a short time but I feel more comfortable with the free version of your website than I would any other WP security premium version.

Nice catch. It's a bit of a basic, not checking that a user supplied value matches a list of what you're expecting, in this case the still weak (in the context of 27% of websites) sha1. With a botnet the size used for mirai then sha1 wouldn't take long to deal with given the weaknesses in it.

If DNSSEC were more wide spread then the issue of code signing could be dealt with relatively easily - certificate/public key stored in the DNS zone. Sites check the downloaded update against the key, sorted. As the authentication of the key is handled in DNSSEC - assuming their DNS zone is kept secure, but they'll all fall down on that as if it wasn't then api could be redirected anyway. It still isn't an insurmountable problem though even without, if the desire is there.

Simply awesome 'detective work'. Well done! And yes, hopefully both the WP community and the WP core team will introduce some form of digitally signing plugins, not unlike, say, Apple does with Mac OS X installations.

Great job for the team!
If I am the one who have the power to infiltrate 27% of the websites on this earth, its great temptation that hard to say no! What you guys did show us how strong your attitude on this. Do really adore you guys and respect :)
BTW, just curiosity come up Mark. If some one alter DNS record of victim's network and create another aliases for api.wordpress.org, will it possible to do this scenario?

I mean its possible to create fake api.wordpress.org and fake github if we can compromise DNS record, right?

It wouldn't be feasible since the request to api.wordpress.org is made over SSL, and WordPress does validate the certificate (which an attacker would not have). It's also worth mentioning there's discussion of bypassing WordPress's SSL verification over on r/netsec: https://www.reddit.com/r/netsec/comments/5e7222/wordpress_download_server_is_a_single_point_of/dab5y08/

Is this an advisable course for me to set? I'm good at checking themes and plugins out in advance, but I wouldn't know if something were infected unless it made the news, or my host, my anti-virus Webroot, Patchman, Wordfence, or Sucuri notified me of the fact. After installing WP, plugins and themes I tend to update everything ASAP as my main defense.

Yes I'd still enable auto-update. This vulnerability has been fixed. However we are trying to encourage WP to improve the update process to avoid any future vulnerabilities like this from being exploited.

With great power comes great responsibility but we can't make sure all the loopholes are fixed. After-all WordPress is developed and supported by the open-source volunteers (mostly). And total security? It is just an Illusion. Almost everything in the world is vulnerable. True security from everything cannot be achieved at all.

Well, coming back to the point...Kudos to Matt Barry and the whole Wordfence team for responsibly disclosing the bug. Keep rocking.

How does this update to api.wordpress.org affect older versions of WordPress? I have a client who (before bringing me on board) had website designers who made changes to the core WordPress files and core theme files. Due to fears of losing functionality that has been baked into the core, they have not allowed any updates to their theme or WordPress core in years! There have only been a small number of automatic updates of sub-versions over the past year. Does this correction to the api.wordpress.org core protect them from this vulnerability, or has only the most recent versions of WordPress benefited from this update? They are at version 3.8.16. I know that by refusing to update, they're vulnerable to a host of issues. I am specifically interested in this issue.

Thinking more about this, why am I not surprised? Wordpress has never inspired confidence about security. I'm still using a plugin to obfuscate my login URL, for example. That should be built into the software.

It is not far fetched to think that if 27% of the internet is built on one software package, this is an issue of national security. Matt and his army of volunteers run the risk of having this all taken away from them unless they demonstrate more diligence. They should have discovered and fixed this vulnerability themselves. If a white hat third party did it, that's just one step away from a black hat having done it. Incredibly serious. We dodged a bullet, clearly. The next bullet is probably in there, waiting to be discovered by a thousand coders sitting in a building in Ukraine, who's job it is to break stuff.

I also find it kinda telling that WPTavern hasn't touched this story at all. They're usually one of the first to pile on and exploit someone else's pain when there's even a hint of something to gain eyeballs over.

Yes it probably was. We automatically delete support requests. Our support team does not provide support in blog comments. At the bottom of this page you'll find the various ways you can get customer support:

In the case discussed in the article, the WP host, api.wordpress.org, was at fault. It could have opened up thousands of sites to hackers, not just a few. But hackers are always looking for these opportunities. I stumbled upon this article when I was doing research for my new book, "Blogging Is Murder", where one of the characters, Liz, “invited” the hacker in because she wasn’t cyber security-conscience enough, it could have easily have been something like this that allowed Connie, the hacker, access to the site so that she could hijack it. You can read the first chapter of "Blogging is Murder" at www.gilianbaker.com.

How long does it take me to hack the average WP (.org) site? 5 minutes.

How long does it take me to hack the average WP (.org) site so that they don't realize they've been compromised, in which case I own them? That's an indefinite period of time, and only worth it if I know the target has a payout for me.

That's the main difference between a professional hacker and a talented scripter (I would say "kiddie scripter" but that seems a bit insulting to some very talented hackers whose only real agenda is to demo their skillz)

While I'm using the 1st person here, this reflects on the only time our servers have been compromised- Nov. 9th this year- affecting sites that were installed but not built yet, and therefore not updated. Some of them had not been since WP 4.1 and some known security bugs.

In my case I searched for the infected signature online and found 87K hits on Google referencing ~23K unique websites which exhibited the same signs. Maybe that's not 27% of the web, but it is a lot.

Case in point, security is only as good as the Admin, or the attention the Admin pays to the site to keep security mechanisms current.

Some of the sites returned in my search results had their front pages taken over, as mine did with some claim to fame by a "cyber-terror" group.

However, more significantly, there were a fairly broad representation of business sites infected without displaying the obvious symptoms of infection. The one site of mine which had active pages published on it DID NOT have its home page taken over, and some other links (other sites) I chased down to see how they were impacted... I doubt the SysAdmin even realized there had been a breach.