WordPress brute-force attack detection plugins comparison.

Two benchmark tests will be performed against the 5 most popular security plugins whose description indicates that they provide protection against brute force attacks, and our plugin NinjaFirewall (WP edition).

The first test will use ApacheBench, the benchmarking tool part of the Apache HTTP server package. It will be a rather quick but totally brutal attack on the WordPress login page, originating from a single IP.
During the second (and much less brutal) test, a distributed attack against the login page will be simulated, from thousands of different IPs. It will be very similar to one of those large attacks that targeted WordPress sites lately.

The 5 plugins are:

All In One WP Security

Better WP Security

BulletProof Security

Limit Login Attempts

Wordfence Security

The test will also include:

NinjaFirewall (WP edition)

If you aren't familiar with it, this is how it works:Attacker => HTTP server => PHP => NinjaFirewall => WordPress

NinjaFirewall sits between the attacker and WordPress. It can filter requests before they reach the blog. That makes it very suitable for detecting and, most important, for blocking brute-force attacks.

WordPress (no plugins)

This is going to be a very interesting part of this article: testing WP alone, without any security plugin. Does it really need those plugins during an attack? We'll see.

Like NinjaFirewall, Wordfence will ignore traffic coming from private IP addresses, but it does not seem to have any option to turn it off. Its /wp-content/plugins/wordfence/lib/wfLog.php file will have to be edited, and the following 11 lines of code from the isWhitelisted() function to be either deleted, or commented out:

WordPress (v3.6.1):
Default installation. Only one user was created: admin (pass: StressTest)
The 'admin' user is important for the test: some of the plugins do not rely on the IP or an access to the login page, but rather on the user. For instance, BulletProof Security will block only if the user exists and has too many failed login attempts. If the user does not exist, it will simply do nothing at all against the attack.

Benchmarking methodology

The first test will record the number of requests per second and the response time, and the second one, the number of requests per second, the duration, the CPU load, memory usage, MySQL queries as well as the number of bytes returned by the Victim.

For each plugin, the same procedure will be followed:

Install and configure the plugin.

Delete the HTTP access log.

Restart Nginx, MySQL and PHP-FPM.

Get the current total number of queries from MySQL (SHOW STATUS WHERE variable_name='Queries';).

Run the sar command to collect system activity information every 5 seconds (sar 5 -rq).

Start the attack from the Attacker server.

Stop the sar command after the attack.

Get the new total number of queries sent to the DB using the above SQL command.

Collect the HTTP access log.

Disable and uninstall the plugin.

To ensure accuracy, each test will be performed 3 times.

First test

ApacheBench will run from the Attacker server with the following command:

Each request simulates an attempt to log in to the admin console. It includes all mandatory fields: WordPress wordpress_test_cookie cookie, credentials (user: admin, pass: bruteforce) and the POST payload inside a postdata.txt file:

It will try to make 50,000 login attempts but, if the attack takes too long, it will stop after 60 seconds. 5 concurrent requests will be used. That will be quite brutal but will match the definition of a stress test:

The goals of such tests may be to ensure the software does not crash in conditions of insufficient computational resources (such as memory or disk space), unusually high concurrency, or denial of service attacks.

The ApacheBench results can be downloaded at the bottom of this article.

The first graph shows the number of requests per second (RPS) that each plugin handled during the attack. The higher, the better:

At a rate of 1,520 requests per second, NinjaFirewall is the clear winner. In order to see the other results, we need to graph them separately:

They all processed the attack* at a rate of less than 30 RPS, BulletProof Security (whose 3 benchmarks show a very high standard deviation) and Wordfence Security being the two slowest. Note how WordPress alone outperformed 4 plugins.

This second graph shows the response time. This is the time, in milliseconds (ms), it took for the server to process the request and send a reply. Obviously, small numbers are better:

With a 3ms response time, NinjaFirewall crushed its competitors (190ms on average).

Second test

This will be a distributed attack attempting to hack into the WordPress login page. It will be much less intensive than the previous one, hence very realistic.

According to the Brute Force Attacks Build WordPress Botnet article from the Krebs on Security blog, the last big attack against WordPress sites was carried out by 90,000+ IPs (servers and/or infected home computers). The blog's article even includes a copy of the username/password list that the attackers may have been using for the attack. It contains 2,927 lines, with quite a lot of duplicates.
It will be used for the attack.

Simulating a large botnet is very easy, thanks to Nginx, by adding those 2 lines to its /etc/nginx/nginx.conf file:

set_real_ip_from 172.16.0.200;
real_ip_header X-Forwarded-For;

This tells Nginx to use the IP from the X-Forwarded-For header of each HTTP request coming from IP 172.16.0.200 (the Attacker). Nginx will forward it to PHP and then, to WordPress and its plugins.

The following simple Perl script will be used to run the attack: it will read the username/password list and will try each of the 2,927 combinations. Parameters, values and cookies are the same as those used for the first test.
Line 29, it generates a random IP for each request and adds it to the X-Forwarded-For field.

The first graph shows the number of requests per second (RPS) during the attack. High numbers are better:

At a rate of 154 RPS, NinjaFirewall is again way ahead of all other plugins. Interesting enough, WordPress (without plugins) comes second with 11 RPS, more than twice as fast as Login Limit Attempts and Wordfence Security.

The next graph shows the total time it took, in seconds, to handle the whole attack. Small numbers are better:

It took NinjaFirewall 19 seconds to get through the whole attack, while WordPress (without plugins) required 276 seconds. Way behind, Limit Login Attempts took around 9 minutes (529 seconds) to perform the same work, and Wordfence Security exactly 625.13 seconds.

Because NinjaFirewall processed the attack in only 19 seconds, attempting to measure its impact on the server load would be irrelevant.
Limit Login Attempts and Wordfence Security, once again, scored poorly by increasing the load much more than any other plugin.

The following graph shows the total amount of RAM, in megabytes, used by each plugins. The lower, the better:

This graph is interesting because it shows that Better WP Security and Wordfence Security used less RAM than WordPress without any plugins (43Mb vs 62Mb). With respectively 82Mb and 96Mb, BulletProof Security and Limit Login Attempts are far behind.
Here too, it would be irrelevant to attempt to graph the amount of RAM used by NinjaFirewall 19-second test.

Next graph displays the total number of queries (read/write) sent to the MySQL database during the whole attack. No query at all, or a low number, is better:

For better performance, NinjaFirewall does not send any query to the database during an attack. WordPress ranks 2nd with 32,234 queries sent (11 queries/POST request). Things start getting worrisome for Better WP Security with 56,461 queries, and totally critical for Wordfence Security which sent no less than 118,096 queries (40 queries/POST request). Looking at the database showed that Wordfence added 6,056 rows to the wp_options table.

The last graph shows the total amount of kilobytes returned by the Victim during the whole attack. The lower, the better:

NinjaFirewall returned a total of 142 Kb (44 bytes for each blocked request) while all other plugins returned around 9 MB (3 Kb/request), the same amount of data returned by WordPress without any plugin.