Pentesting Web Services with Proprietary Formatted Input

Submitted by abb on 29 September, 2011 - 14:34

Introduction

From time to time I come across a web service that expects its input in some proprietary format, usually JSON distorted in one way or another. A vulnerability scanner knows nothing about that stuff and can't properly fuzz it. (At the time of this writing Acunetix and Burp Pro support JSON only in HTTP responses.) In this case one has to resort to pure manual testing or partially automatic test with a fuzzer. Both approaches have their limitations, and I decided to finally find a way to run an automated scanner against proprietary web services.

In this blog post I will talk about how this problem can be approached. It includes the description of a practical setup, which I used twice to pentest proprietary web services with Burp (one expecting JSON-formatted input in the body, another in a POST parameter). It can be adapted for any other input format.

The described setup has a lot of limitations (see note 3 in the end of this document) and is somewhat cumbersome. Unfortunately, at the moment it seems to be the only way to approach testing the wheels being reinvented by developers over and over again.

Burp receives the response and possibly forwards it back to the client

Our problem is step 3. Burp cannot properly fuzz the request it can't understand. To let it do so, we need to translate the original request into something Burp can understand and fuzz, and then translate it back.

Altered request flow

To achieve the above, the request flow has to be changed, for example as following.

The client resolves the host name (from URL) trying to get the IP address of the WS (web service), but instead arrives to our intercepting proxy. I use Burp Pro set to listen on port TCP/8080 with redirection to localhost:80.

The client sends HTTPS POST with JSON-encoded body. For example, consider the following HTTP request coming from the client.

Burp receives this HTTP POST and forwards to Apache listening on localhost:80 and set to redirect everything to a custom PHP script called http2tpl.php. The following addition Apache vhost does the redirection:

http2tpl.php script transforms HTTP POST request into plain HTTP GET with parameters, which can be parsed and fuzzed by scanners and sends this request back to Burp (localhost:8080, hardcoded in the script).

Now Burp received a request with all the parameters exposed. It can fuzz them if asked to do so. Next Burp, being a proxy, sends the request back to Apache on localhost:8080, but now it goes to another PHP script.

When tpl2http.php script is invoked, it re-assembles the original request, possibly with some parameter values altered by scanner.

After the request is assembled, tpl2http.php script forwards it to Burp again, but to another listener set to forward the requests through to the target server. The port the second listener runs on is hardcoded into this script (I use TCP/8081). This resulting request will looks very much like the original one (it is slightly different from the original because PHP JSON decode/encode functions drop redundant white spaces). It will be proxied by Burp towards the target server.

{"id":2,"jsonrpc":"2.0","method":"get***History","params":{"params":{"startIndex11952520' or 1=2--":0,"maxResults":5}}}

Summary

Here is the summary of the test setup for HTTP services:

The first Burp listener (:8080) set to redirect requests to local Apache (:80)

The client set to use this listener for HTTP requests.

The second Burp listener (:8081) is set to invisible mode

Local Apache (:80) set to redirect all requests to http2tpl.php

Variables proxyhost/proxyport in http2tpl.php are set to the first Burp listener localhost:8080 (default)

Variables proxyhost/proxyport in tpl2http.php are set to the second Burp listener localhost:8081 (default)

http2tpl.php and tpl2http.php are deployed in Apache's webroot.

If HTTPS needs to be supported:

The third Burp listener (:8443) set to redirect requests to local Apache (:443)

The client set to use this listener for HTTPS requests.

The fourth Burp listener (:8444) is set to invisible mode

Local Apache (:443) set to redirect all requests to http2tpl.php

Variables $proxyhost/$proxyport in http2tpl.php are set to the first Burp listener localhost:8080 (default)

Variables $proxyhostssl/$proxyportssl in tpl2http.php are set to the fourth Burp listener localhost:8444

http2tpl.php and tpl2http.php are deployed in Apache's webroot.

Notes:

The same approach can be used to handle other input formats by changing the implementation of http2tpl.php (see comments inside). Script tpl2http.php is not JSON-specific, it merely gets HTTP request line, headers, body template, and set of parameter substitutions it needs to make and assembles the request back.

The sample requests above are simplified, real GET /tpl2http.php requests carry serialized HTTP request line, headers, and body template. These values are sent in DST parameter, its integrity protected with HMAC in DST_SIGN parameter (to make sure Burp can't mess with them).

The current implementation is extremely ugly, and probably does not let Burp identify all vulnerabilities it could have identified. It breaks binary content, for example images. This is because it uses fgets() function of PHP, which drops \x0d characters. It does not correctly relay HTTP headers. Still, it is better than nothing and has already helped me to find some input validation flaws.