Wednesday, November 4, 2009

Web servers use server-side scripting engines to support web applications. Using the web server configuration, dynamic pages are identified by filename extensions and processed by their relevant processors.

Below is an example of IIS 6.0 application configuration. As you can see, asp files are processed by asp.dll and php files are processed by php-cgi application. Other web servers also use similar methods.

In some situations, web servers respond differently to nonexistent files with known extensions. Sending requests to random filenames with known extensions and comparing the HTTP response results may reveal the server-side scripting technologies supported by the web server.

While scanning customer networks, I saw that various web servers can respond differently to many known extensions including asp, aspx, cfm, php, jsp, shtml. Also additional vulnerabilities can be found in these responses, including Internal IP address disclosure, application errors etc.

You can see some real life examples below. We will send a simple request to nonexistent asp, html, and php files. Then compare the responses. These tests will be made to the root directory, but you should also notice that some special subdirectories’ configuration might be different.

As you can clearly see, the web server responds to nonexisting files as 302 found message and redirects us to /search/error.html. But when it comes to PHP files, we see a redirection to /forbiddenip/forbidden.html

Website responds differently to nonexistent php files. So, X-Powered-By PHP header seems valid. Since URL rewriting is in place, we can clearly verify that the wiki application used by nginx is written in PHP (It is actually MediaWiki).

I call this method as technology detection by using known filetypes. Using this method in addition to other fingerprinting techniques, such as HTTP response banner grabbing, is useful to improve web security scanners.

I previously implemented this in Arachne, a simple web security scanner that I developed for my MSc thesis back in 2006. It sends requests to nonexistent files with known extensions, then compares the results to see if a technology is used in that web server.

With technology detection by using known filetypes;

Web application scans can be optimized for detected technologies

Web application scanners can reduce the number of tests performed

Scanners can reduce false-positives for nonexistent files

If URL rewriting is used, this method can be used to determine used technologies in the web application