Wednesday, July 1, 2009

Objective

I will present a general solution to the two-way compression of WebServices" issue. The issue can be critical in a complicated SOA solution where a lot of data is passed not only from the server to the client but also from the client to the server. The solution is a part of a Data Service component I currently work on.

Motivation

On one hand, the application server can compress the HTTP data sent to the client "out-of-the-box", just enable "HTTP Compression". While such solution seems attractive, it does not provide two-way compression. It's nice to have a server's response compressed, however uploading huge amount of data is still a problem.

On the other hand, a generic solution already exists and consists in writing a custom SOAP extension. The solution by Saurabh Nandu dates 2002 is described here. There are however two issues with that approach. A small issue is that you have to apply a custom attribute to both server and client code. While applying a custom attribute to a server method is not an issue, the client proxy class is regenerated each time you update the reference which means that you have to remember to manually edit the proxy class after you update the reference. Unfortunately, there's also a big issue. We've observed that the solution causes random OutOfMemory exceptions in a production server environment! The randomness was a complete disaster for our proprietary software!

Towards the solution

Let's start with a basic code we'll enhance during this tutorial. Open Visual Studio, create a new solution with two projects: a console applications and an ASP.NET Web Service application.

Implement a method on the WebService.

using System;

using System.Collections.Generic;

using System.Linq;

using System.Web;

using System.Web.Services;

namespace WebService1

{

[WebService( Namespace = "http://tempuri.org/" )]

[WebServiceBinding( ConformsTo = WsiProfiles.BasicProfile1_1 )]

[System.ComponentModel.ToolboxItem( false )]

publicclass Service1 : System.Web.Services.WebService

{

[WebMethod]

publicstring HelloWorld( string Request )

{

return Request;

}

}

}

Note that the method has an input parameter and just returns it's value to the client. We are going to pass really long strings here and we'll gonna see how much data is actually passed to and from the server.

In a console application, add a web service reference and following code:

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

namespace ConsoleApplication60

{

class Program

{

staticvoid Main( string[] args )

{

MyService.Service1 s = new ConsoleApplication60.MyService.Service1();

s.Url = "http://localhost.:3112/Service1.asmx";

StringBuilder sb = new StringBuilder();

for ( int i=0; i < 10000; i++ )

sb.Append( "qwertyuiopasdfghjklzxcvbnmqwertyuiopasdfghjklzxcvbnm" );

Console.WriteLine( s.HelloWorld( sb.ToString() ) );

Console.ReadLine();

}

}

}

Note that I manually set the Url of the WebService and what I actually did was to add a dot after the "localhost". This is to be able to use a HTTP sniffer, Fiddler.

Run the application and inspect the HTTP session in Fiddler. Please take a look at "Content-Length" headers for both the request and the response, the content length of the request is 520318 bytes, and the content length of the response is 520352 bytes.

One way compression (compression of the response)

The OutOfMemory exceptions caused by the solution based of a custom SOAP Extensions made us to search for another approach and lead us to a common and well known "HttpCompressionModule on a server - HttpWebResponseDecompressed on a client proxy". This solution solves only the half of the compression issue - the data sent from the server to the client is compressed.

Start with a HttpCompressionModule: add a HttpCompressionModule.cs file into the WebService project:

Now go back to the client console application and add two methods to the partial proxy class. Assuming that the proxy's class name is MyService, add MyService.cs:

publicpartialclass Service1

{

#region WebResponseCompress

protectedoverride WebRequest GetWebRequest( Uri uri )

{

HttpWebRequest request = (HttpWebRequest)base.GetWebRequest( uri );

request.Headers.Add( "Accept-Encoding", "gzip, deflate" );

return request;

}

protectedoverride WebResponse GetWebResponse( WebRequest request )

{

returnnew HttpWebResponseDecompressed( request );

}

#endregion

}

Note that what this does is to make sure that the correct header is sent to the server and that the response is actually decompressed on the client side. The HttpWebResponseDecompressed is a common class but it's not in the Base Class Library so here it is:

publicclass HttpWebResponseDecompressed : System.Net.WebResponse

{

private HttpWebResponse response;

public HttpWebResponseDecompressed( WebRequest request )

{

try

{

response = (HttpWebResponse)request.GetResponse();

}

catch ( WebException ex )

{

response = (HttpWebResponse)ex.Response;

}

}

publicoverridevoid Close()

{

response.Close();

}

publicoverride Stream GetResponseStream()

{

if ( response.ContentEncoding == "gzip" )

{

returnnew GZipStream( response.GetResponseStream(),

CompressionMode.Decompress );

}

elseif ( response.ContentEncoding == "deflate" )

{

returnnew DeflateStream( response.GetResponseStream(),

CompressionMode.Decompress );

}

else

{

if ( response.StatusCode ==

HttpStatusCode.InternalServerError )

returnnew GZipStream( response.GetResponseStream(),

CompressionMode.Decompress );

return response.GetResponseStream();

}

}

publicoverridelong ContentLength

{

get { return response.ContentLength; }

}

publicoverridestring ContentType

{

get { return response.ContentType; }

}

publicoverride System.Net.WebHeaderCollection Headers

{

get { return response.Headers; }

}

publicoverride System.Uri ResponseUri

{

get { return response.ResponseUri; }

}

}

Now run the solution and inspect it with Fiddler:

Note that the content length of the server's response is not only 5771 bytes! (Note also that the 100:1 compression ratio is rather unusual and is caused by repetitive data I send to the server from the console application!)

Two-way solution (compression of requests and responses)

We are now ready to enhance the partial solution and enable a two-way compression (and this is my contribution to the issue).

First, go to the HttpCompressionModule and add a line:

...

void context_BeginRequest( object sender, EventArgs e )

{

HttpApplication app = sender as HttpApplication;

HttpContext ctx = app.Context;

if ( !ctx.Request.Url.PathAndQuery.ToLower().Contains( ".asmx" ) )

return;

if ( IsEncodingAccepted( "gzip" ) )

{

/* INSERTED LINE HERE!

We add a filter to decompress incoming requests.

*/

app.Request.Filter =

new System.IO.Compression.GZipStream(

app.Request.Filter, CompressionMode.Decompress );

app.Response.Filter = new GZipStream( app.Response.Filter,

CompressionMode.Compress );

SetEncoding( "gzip" );

}

elseif ( IsEncodingAccepted( "deflate" ) )

{

app.Response.Filter = new DeflateStream( app.Response.Filter,

CompressionMode.Compress );

SetEncoding( "deflate" );

}

}

Then, go back to the client application, and modify the partial class definition:

I'm trying to implement the solution here, but I'm having an issue with the GZipStream request side. I did some packet sniffing and noticed the request is GZip encoded, but it's the same size as my non-encoded request? Has anyone run into this issue?

@Rubem Rocha: the easiest way would be to add a stronger condition to the first "if" of the BeginRequest method of the compression module. Just check if the url contains ".asmx" but DOES NOT contain "?wsdl".

In Web Services:1. Add the class(HttpCompressionModule) to webservices project 2. Add the line in the web.config

And In Client:1. Add Two Classes HttpWebResponseDecompressed and HttpWebRequestCompressed in the LCMS Client2. Make changes in the Partial class of the webservice with the same name as present in the reference.cs and override the GetWebRequest Method.

It is very nice solution, hoever I have few notes:1. It is not workng when BufferResponse is set to false2. If I would like to disable it temporary on server the client will stop work, which is a bit dangerous.

This is very much satisfied by providing the nice understanding and implement the different technology is visible in this blog. Thanks a lot for sharing the nice implement and providing the great info.Web developer

I found several bugs with this. It seems to try and unzip the request unconditionally (when the request was not actually GZIP'd). It also does not maintain the content-encoding header if an exception occurs on the webservice. In order to fix these errors I had to make a few changes.

I've posted an updated version of the code here: http://pastebin.com/Aak9FUiw

Hi, thank you for such wonderful codes. I am facing some problem when trying to consume the webservice from my console app, I have got an error saying "Response is not well-formed XML." Any idea what could causing this?

hi, I manage to solve my problem above by enabling web service extension for ASP.NET 4 (previously only enable for ASP.NET 2), but however when I check with Fiddler, the compression ratio is 26:1, as contrast to 100:1, i use back the same string, what possible cause for this compression ratio difference?

@HonWaiLai: I would not be concerned by different compression ratio. This heavily depends on actual data and the implementation of the compression algorithm. You run it on .net 4 while my tests were performed on .net 2.

About

I am a software architect and an academic lecturer. I've been awarded Microsoft MVP in C# Architecture between 2005 and 2010. Got a PhD in computer science in 2008.
Currently I work mostly with C# and Javascript and love both languages. I consider myself a design/architecture patterns evangelist.
In 2003 I wrote first Polish book on C#/Windows programming, "Windows oczami programisty" (Windows through the eyes of a programmer) (the book is out of print)