In this continuingfive-article series, I share insights and discoveries concerning website security and protecting against malicious attacks. In this fourth article, I build upon previous ideas and techniques by improving the directives contained in the original, 2G Blacklist. Subsequent articles will focus on key blacklist strategies designed to protect your site transparently, effectively, and efficiently. At the conclusion of the series, the five articles will culminate in the release of the next generation 3G Blacklist.

Improving the RedirectMatch Directives of the Original 2G Blacklist

In the first version (2G) of the next-generation blacklist, a collection of malicious attack strings were compiled from access logs, blocked via Apache’s RedirectMatch directive, tested for effectiveness and transparency (i.e., non-interference), and finally converted into the 2G Blacklist. This 2G Blacklist continues to serve as my primary line of defense here at Perishable Press, replacing several other lists, including a million individual IP addresses and even the Ultimate htaccess Blacklist itself. So far, the results have extraordinary.

Of course, as discussed in Building the 3G Blacklist, Part 3, static, “fix-it-and-forget-it”-type strategies ultimately fail when dealing with the perpetually evolving nature of the enemy (i.e., malicious attackers, crackers, and other scumbags). Thus, as stated in the original 2G article, the blacklist is a work in progress, designed for periodic updates as new attack patterns and trends are identified. In the remainder of this article, we consider various aspects of the previous blacklist and consider various improvements. Note that it is not necessary to assemble and transcribe any of the new blacklist directives presented herein. At the conclusion of this series, each of the different strategies will be combined into a single, comprehensive security strategy: The 3G Blacklist!

Miscellaneous Character Strings

Besides query strings, miscellaneous, random, and unusual character strings are perhaps the most frequently observed characteristics of common server attacks. For example, each of these sequences is a common sight to those of us concerned with site traffic and security:

/,

//

f-.

...

Yet these strings are missing from the previous version of the blacklist. In the new version, we block these character strings, as well as several others:

:

;

<

>

.inc

alt=

ftp:

ttp:

.$url

/$url

/$link

This is powerful stuff. Any URL request containing one (or more) of these character strings will be denied via an immediate 403 Forbidden error. Attackers frequently target server-side includes (.inc), which are protected here in the fifth line. Inclusion of various hypertext protocols are also common, and are blocked here in the third and fourth lines. Another common attack vector involves the ubiquitous $url variable (often recorded as \\\'.$url.\\\'), which is also protected in this set in the last two lines. Compared with the equivalent section of the previous blacklist,

.inc

alt=

http://

..the new version is much improved, covering a far more expansive field of vulnerability using only a few additional lines.

PHP File Types

In the previous blacklist, common PHP file targets were added to the list in ”shotgun”-like fashion. While sorting out the madness, readers pointed out several conflicts related to the blacklist:

In addition to correcting these discrepancies, PHP-related expressions that are highly specific to a particular platform or context (e.g., admin.webring.docs.php) have been generalized to match any file type. Conversely, PHP-related expressions that are more general in application (e.g., about.php) have been left unaltered. Further, several new PHP file names have been added, some have been replaced, and some have been removed.

For the record, here is the PHP-related section as it appears in the previous, 2G Blacklist:

Directory Path Segments

Also new in the 3G Blacklist are several common directory path segments. These directory names appear frequently in malicious attacks, even though they have no business being requested directly via http protocol, especially by someone trying to hack your site. ;)

Certainly straightforward. The \/ on either side of these names ensure that only directories are matched. Including these terms in post titles is completely safe. As an aside, I may end up removing \/path\_to\_script\/ in the 3G Blacklist. If you think it should remain, drop a comment and explain. Likewise, if you see potential conflicts with any of the other expressions, now is the time to let me know.

Miscellaneous Functions

Another commonly observed component of malicious attacks includes a variety of miscellaneous functions appended onto the end of URLs. The previous, 2G Blacklist does not include such functions, but I finally got sick of seeing the little bastards squirming their way into my access logs. Thus, the new blacklist blocks any of the following:

Lastly, I will mention the removal of eleven esoteric, randomly generated character strings that are no longer necessary. These were most likely one-time vulnerability probes no longer in service. Here are the dropped entries:

And finally, the redirectmatch directives as written in the 2G Blacklist have been rewritten to adhere to accepted standards:

OLD:redirectmacth

NEW:RedirectMacth

A subtle improvement, perhaps, but an improvement nonetheless!

Wrap Up..

With this “sneak peak” into the rules comprising the upcoming 3G Blacklist, we enjoy the opportunity to suggest changes, add directives, and improve the overall strategy before the final release. The changes discussed here represent a majority of the improvements made involving RedirectMatch directives, however there remain several other changes that will be included in the upcoming release of the 3G Blacklist. Also, if you see any potential issues or conflicts (or worse) involving any of the code presented here, please speak up and leave a comment. Thank you!

Hi Sat :) Thank you for the great feedback. As discussed in part 3 of the series, I have found that excessively large blacklists will in fact reduce the performance of your site. Of course, the degree to which this occurs depends on many environmental factors, such as available server memory and the amount of traffic you happen to be receiving. In other words, a site with excellent performance without insanely large htaccess files will probably still do well with them. Conversely, sites with mediocre or poor performance to begin with should probably try to keep the amount of htaccess processing to a minimum.

Personally, I am running this site on a shared server that usually performs well, however I have been doing everything possible to maximize performance. In the process of optimizing, I did notice a improvement in page loading times after I had removed several large blacklists that had been piling up. I enjoyed the performance boost so much that I began to develop alternate methods of blacklisting scumbags that don’t require hundreds upon hundreds of htaccess operations.

Discovered a Bug??
Im way over my head…
Far from an expert….
with a bit of trial and error gotRedirectMatch 403 // good
BUT using phpnuke, the avatar on the front home page login disappeared yet appeared everywhere else
I found the url to the avatar for some reason had a // in it
Since the hack stuff is =http:// changing the code toRedirectMatch 403 ://
fixed it
ALSO have come across another hack form/modules.php?set_albumName=http%3A%2F%2Fwww.winbd.net%2Fadmin etc etc=http% being the common denomiator

Hi Steptoe,
Nice work! I think changing the \/\/ to \:\/\/ as you have done is an excellent solution. Granted, a few invalid requests containing the double forward slashes may get through, but the majority of them will still be stopped cold. Even so, I would be interested in seeing an example of the phpNuke URL string that contains the double slashes. Such character sequences are commonly employed in query strings, but to place them in the root portion of the URL seems odd..

As for the encoded transfer protocol, that is indeed a good common denominator. Currently, the query-string section of the 3G Blacklist blocks any URL containing http:, which accounts for a large majority of attempted exploits. However, I have also seen a few URL-encoded versions slip through (i.e., http%3A, etc.). I think blocking these URL-encoded query-string protocol requests is an excellent idea. I will be adding the following directive to the second section (query strings) of the next version of the 3G Blacklist:

Projects

About the site

Perishable Press is the work of Jeff Starr, professional developer, designer, author, and publisher with over 10 years of experience.
Check out some of Jeff's books and projects, follow on Twitter, or learn more »

Fun fact: Perishable Press has been online since 2005, and features over 800 articles and more than 11,000 comments. More stats »