Introduction

Many web applications such as proxy web servers and multiple search engines are required to access the HTML pages of other websites. The classes WebClient, WebRequest, and WebResponse are usually used to perform these requirements in ASP.NET.

On the other hand, a WebBrowser control is used in a Windows Forms application to browse web pages and other browser-enabled documents. WebBrowser provides many events to track data processes, and many properties and methods to access and create new contents on the HTML element level.

Are there advantages to using a WebBrowser control in an ASP.NET application? And is it possible? The answer is yes. This article and the example prove this claim. This example is more complicated than a simple web server that probably retrieves and browses only one web page when processing a client request; it uses a WebBrowser control to log in to the Microsoft website of www.live.com, waits until the content is stable, and then retrieves the HTML content, cookies, the values of input elements, and the script variables. The example discovers at least five advantages to using a WebBrowser control in ASP.NET:

Browses HTML pages while debugging;

Handles cookies automatically without related coding;

Allows scripts running and changing HTML contents. Conversely, it is impossible for the WebClient, WebRequest, and WebResponse classes to support script running;

Accesses HTML elements and script variables programmatically;

The HTML contents in a WebBrowser control are always flat, but that in WebClient or WebResponse may be compressed.

Notice that WebBrowser runs on the web server-side, not on the client-side.

The sequence diagram below illustrates the overall concept of the article and the relationships between the major parts. This diagram and the flow process table at the bottom are helpful for reading the article.

Background

The technique of using the WebBrowser control in a web server server-side has been successfully used in a real project that I worked on, which searches pubmed.gov, analyses and reformats the response with data from other resources, and displays them in a new webpage. I also had done some web server projects by using WebRequest and WebResponse. I felt that using the WebBrowser control is much easier than using WebRequest and WebResponse.

Using the Code

Download the source code and unzip it to a folder such as <WBWebSitePath>.

Run Visual Studio.

Click File, Open, and Project/Solution..., choose and open <WBWebSitePath>/WBWebSite.slu for using Visual Studio 2005;

Or, File, Open, and Web Site..., choose and open <WBWebSitePath> for using Visual Studio 2008.

Click the toolbar button Start Debugging (F5) to run the example.

On the checkbox "Display form and WebBrowser control",

Check it to display a form that contains a WebBrowser control;

Or uncheck it to hide the WebBrowser control so its container form will not be created.

Type an email address, such as <name>@hotmail.com or <name>@live.com, in the Email Address text box; type a password in the Password box.

Click the Submit button to submit the login request to the web server.

Wait for the response from the web server.

Optionally, set a breakpoint in getHtmlResult() and/or other methods of the file IEBrowser.cs to access the HTML elements using the various methods and properties, such as Document.GetElementById(), GetElementByTagName(), HtmlElement.InnerHtml, and GetAttribute().

Optionally, click "Sign out" in the WebBrowser control to logout from http://www.live.com. Thus, you can login again.

Optionally, login with a wrong email address or a wrong password to see the response.

Operation and Response of the Web Server

While receiving the login request, the web server creates a WebBrowser control and navigates it to http://login.live.com, sets the input elements of the email address and password in the page with the values from step 5, and submits the form, to which the input elements belong, programmatically.

A result page is responded from the URL in the form's action attribute to determine if the login process is successful, and is loaded in the WebBrowser control. The WebBrowser control exists on the machine where the web server is, and it is visible if the checkbox in step 4.1 is checked. The URL of the result page is displayed in the title bar of the form.

Response 1 - The web server retrieves the result page from the WebBrowser control, saves it as a local HTML file, and then calls the JScript window.open() method to browse it in a new Internet Explorer browser instance. The file address is displayed in the address bar of the browser. I do not use the HTML IFRAME tag to browse the result page in the original Internet Explorer browser instance, since the JScript code of the page always checks and tries to reload the page into the top window.

Response 2 - The web server analyses and retrieves the values and types of the JScript variables, HTML input elements, and cookies of the result page from the WebBrowser control, and responds to them as HTML tables. If the type of a JScript variable is object, its children are responded to; if a child is still an object, the string "[object]" is responded. The navigation number here is the number of WebBrowser controls raising the Navigating event during login (see explanation).

Technical Specifications

Conditions to Run WebBrowser in ASP.NET and Solutions

First of all, a WebBrowser control has to be in a thread set to single thread apartment (STA) mode (see MSDN), so I need to create a thread and call the SetApartmentState() method to set it to ApartmentState.STA before starting it.

Then, as a Windows Forms control, a WebBrowser control needs to be in a message loop for raising events continuously (the Navigating and DocumentCompleted events are used in the example); otherwise, it only raises the first event. There are two options to create a message loop: calling System.Windows.Forms.Application.Run(System.Windows.Forms.Forms form) or calling System.Windows.Forms.Application.Run(System.Windows.Forms.ApplicationContext applicationContent). I chose the second because a Windows Form is not necessary for a WebBrowser running in ASP.NET.

The third condition and its solution are based on three threads, and the relationships among them are as below:

The main thread is the executer of the ASP page event handles, such as Page_Load() and Button_Click(), in the code file of the ASP page. The thread has to exist until it receives the result from the third thread and writes the result into its page content. Thread.Join() and AutoResetEvent can be used for the synchronization requirement, and I chose the latter. Another possible solution is AJAX - the main thread exits, but AJAX in the page will send interval requests to check and update the page content.

The second thread, which the WebBrowser control is in and has been introduced above in the first and second conditions and solutions, is created by the main thread, and is of both states of STA and ApplicationContext. This thread will exit after the third thread is created. If this thread was blocked, neither are the events in the third thread raised nor are the event handles called.

The third thread processes the message loop, and so it executes the WebBrowser event handles such as Navigating() and DocumentCompleted(), and the C# callback function. It is also responsible for retrieving the contents of the page in the WebBrowser control and setting AutoResetEvent to allow the main thread to continue.

Functionality to Ensure the Content in the WebBrowser is Stable and Complete

A simple website may respond to a client request only on one page, so the content can be directly retrieved in the DocumentCompleted() event handle. The login process of www.live.com, however, is very complicated since the remote web server and the WebBrowser control exchange their requests and responses up to eight times before login succeeds. Thus, I need an appropriate functionality to check if the content loaded in the WebBrowser control is stable and complete.

Fortunately, a useful factor is that the times of calling the DocumentCompleted() event handle are always equal to the times of calling the Navigating() event handle. The functionality based on the factor is as below:

Define and initialize a counter.

In the Navigating() event handle, the value of the counter increases by one.

In the DocumentCompleted() event handle, request the script in the active page to call a C# callback function in the third thread, with the value of the counter in an interval time.

In the C# callback function, compare the value set in step 3 with the current value of the counter. If they are equal, it means that the content in the WebBrowser control is stable and complete, and the main thread can continue to run; otherwise, wait for the next call.

The C# callback function is a window.exteral script object (see MSDN), and is defined in a separate class from the one derived from ApplicationContext because [System.Runtime.InteropServices.ComVisible(true)] and ApplicationContext cannot be set on the same class.

Besides using a counter and a callback function, other considerable functionalities in a practical program could be using a timer and comparing the page contents in the WebBrowser control with patterns.

In the C# callback function, WebBrowser.DocumentText is used to get the HTML content, and WebBrowser.Document.Cookie is used to get the cookies, and WebBrowser.Document.GetElementsByTagName("INPUT") and HtmlElement.GetAttribute() get the input elements. The values of these are not simply from the remote website, but also are probably changed during the script running after the page being loaded.

This is a more difficult task than retrieving the values of cookie and HTML input elements because there is no method to get the script variable names and their values directly. I have to:

Use WebBrowser.Document.GetElementsByTagName("SCRIPT") to get the script source code.

Filter out the function definitions, function calling statements, and other useless data from the script source code.

Create a script available name table based on the result of step 2.

Call InvokeScript() to send the name table to the script in the active page in the WebBrowser control, and request the script to retrieve the script variable values and types, and send them back by calling the C# callback function.

The script code is saved in jscript.js. I use a for-in loop to get the names and values of the children of an object.

Avoiding the standard way, which adds the reference Microsoft.mshtml to the project, create a HtmlScriptElement and call AppendChild() to insert the script code into the active HTML page. I send all the script statements together as a string parameter of the setTimeout() script function. I use setTimeout() here because it needs a time delay.

The table below explains the flow process of codes in the example, the relationships between the codes and threads, and the data transfer between the client, server, and the Microsoft web server. The table may help you understand the example easily. The column titles of the table are:

Client-side - the client-side actions

Server-side - the server-side actions

C# Codes -the C# codes of the example

Explain - the explanations of C# codes

File name - the file name that the C# codes belongs to

Thread - the thread that the C# codes belongs to

Microsoft Web Server - the actions of the Microsoft web server

TD - The data transfer direction

Client-side

TD

Server-side

TD

Microsoft Web server

C# Codes

Explain

File name

Thread

Browser displays Default.aspx (screenshot Login page in the article)

<-

Default.aspx

A user checks the checkbox for displaying the WebBrowser control, types an email address and a password, clicks the button to submit the login request.

This is the C# callback function for calling by script. The variable count here has the previous value of navigationCounter in Steps 1 and 2. The owner.<br />navigationCounter is the navigationCounter in Steps 1 and 2. The ASP.NET system will repeat Steps 1, 2, and 3 until owner.<br />navigationCounter equals count.

Are you sure this is going to work if more than one user requests the page at the same time?

In general, code that isn't written for a multi-threaded server environment doesn't work in a multi-threaded server environment. For instance, in a Windows Forms application, it would be reasonable for this code to use static data. But in ASP.NET, those static variables will be used by multiple requests at the same time.

This is probably not what the author of the control intended.

Which means that the result is probably not what the consumer of this control intends, either.

Please test this code under a load which includes multiple simultaneous requests.

Also, anyone trying this technique will have to similarly test their code under a multi-request load.

Very good work, now i need help to transfer cookies in the client browser so client could problably continue navigation by its own: in few words, i try to have a asp.net portal that automate the post of credentials in another site (user and password : for security reason the text may be hidden) but after that, the client must be indipendent. Can you help me? i dont know how make it work:

You'd probably need to invoke scrolling a certain distance (offset) via JavaScript and re-capture each time. You could either just keep scrolling and capturing in a "while" loop, while you still get data, or you could set your loop to do it a certain amount of times. And, of course, you'd have to concatenate your HTML together before displaying.

I am using this code along with some modification to get an image of the web browser's visible window. The problem is that when I try to load a page that has jQuery on it, the jQuery doesn't load/run so the image I get back is messed up. Have you (or anybody else) got this thing to run with jQuery enabled websites?