Building Wireless Web Clients, Part 1: Pitfalls of MIDP HTTP

Wireless Java applications are, by their nature, network-centric. The devices that these applications run on are, however, less predictable. Most notably, the precise nature of the network connection depends both on the device and on the services provided by the network to which it is connected. Some wireless devices may
be directly connected to the Internet, while others are only able to access it through a gateway. Whatever the nature of the underlying network, a wireless Java device that conforms to the Mobile Internet Device Profile (MIDP) specification is required to provide the illusion that it is directly connected to the Internet by implementing the HTTP support that is part of the MIDP Generic Connection Framework API. A description of this framework can be found in the article "Invoking Java ServerPages from MIDlets" by Qusay Mahmoud. The device is not required to do this by including a TCP/IP protocol stack to carry the HTTP protocol messages; it is permitted to use the network's existing protocols as the bearer, as long as it preserves the behavior that an HTTP-based application would expect.

The lack of a TCP/IP stack usually means that access to lower-level programming paradigms, such as sockets, is not guaranteed to be available to a MIDP application, even though the Generic Connection Framework provides an interface to such low-level services, and the next version of the MIDP specification is likely to require their inclusion in all devices. For the time being, then, wireless Java applications will have to use HTTP to communicate with the outside world. However, several features of the wireless Java environment make it slightly more difficult to write a MIDP HTTP client than would be the case with J2SE. This article highlights some of the pitfalls that are unique to the J2ME environment, using an example taken from Chapter 6 of my recently published book, J2ME in a Nutshell.

The Bookstore Web Client

The Web client used in this article is, in principle, very simple. Given the ISBN for a book, we want to connect to Amazon's online bookstore, retrieve the book's details, and display its title, sales ranking, and the number of customer reviews. In the second part of this article, we'll get a little smarter and store these details, along with the book's title, so that we can go back to the site later and get updated details without having to remember the ISBN. For now, our problem will be how to get the information that we need and how to display it to the user.

Before proceeding with the technical details, let's take a look at the completed client in action. To build and run this example, you'll need the source code, which is available in this .zip file, and a suitable development environment. The example source code is appropriately packaged for the J2ME Wireless Toolkit, which can be downloaded for free and includes emulators for several types of mobile devices. This article assumes that you are using this tookit. The source code can, however, be used with other development tools, such as Forte for Java.

Having downloaded the source code, unpack it below the J2ME Wireless
Toolkit's apps directory, start the toolkit's KToolbar application, press the Open Project button, and select the project J2MEHttp. If this project does not appear, make sure that you have a directory called J2MEHttp below the apps directory of the J2ME Wireless Toolkit installation. If you don't, then you have not unpacked the example code to the correct location.

With the project open, press Build to compile the source code, select an emulated device, and press Run to start the emulator. When the emulator starts, it will offer the choice of two MIDlets to run. In this case, chose RankingMidlet; the other one, PersistentRankingMidlet, is the subject of the second article in this series.

Figure 1: The Bookstore Web Client

The RankingMidlet MIDlet presents a form where you can input an ISBN, as shown in the left side of Figure 1. Supply the ISBN of your favorite book and select OK. After a short while (assuming the ISBN is valid), you'll see the title, sales ranking, and the number of reader reviews for your chosen book, as shown in the right side of Figure 1.

So how does this client work? Apart from the details of the user interface, which we're not going to cover in this article, the client does three things:

Connects to the Amazon Web server and requests the store page for the
chosen book.

Reads the HTML page that the server returns.

Interprets the HTML page to extract the details that it needs.

This all sounds simple enough, but there are a few pitfalls waiting to trap the unwary along the way. Let's see what can go wrong by examining each of these three steps in more detail.

Connecting to the Server

The first problem we encounter is how to get the correct HTML page for a book, given its ISBN. It isn't difficult to work this out -- just point your browser at Amazon's Web site, enter an ISBN in the search box on the home page and look at the URL of the page that is returned. If you do this, you'll find that the browser ends up loading a URL that looks something like "http://www.amazon.com/exec/obidos/ASIN/156592455X/102-0259985-4227363", for a book with ISBN 156592455X.

In fact, everything after the ISBN in this URL is concerned with tracking your user session with Amazon, and does not have to be supplied on initial contact. Therefore, to get the details for this book, we only have to make an HTTP GET request with the URL
http://www.amazon.com/exec/obidos/ASIN/156592455X.

This isn't how the browser got the page, however; when you entered an ISBN on the home page, this URL was not constructed directly. Instead, a query was created and sent to the server, which allowed it to return the correct page. The fact that Amazon also recognizes a more explicit URL that gives the same result is useful for this client, but you might not be so lucky if you had the task of creating a client for a less cooperative server. To demonstrate how to handle the more general case, we'll show you how the browser actually fetched the correct page and show you the Java code that achieves the same result.

The search feature on the Amazon home page is implemented using HTML form tags. When it comes to the nitty-gritty detail, the form causes an HTTP POST request to be sent to the URL
http://www.amazon.com/exec/obidos/search-handle-form/0
in which the body of the message contains the query itself, in the form:

index=books&field-keywords=isbn

This query causes a search of all books on the site (as distinct from software, electronics, etc.) for the given ISBN. Recreating this in Java is quite straightforward -- we simply use the Connector class from the Generic Connection Framework to open a connection to the URL shown earlier, set the request method to POST, open an output stream, and write the query to it. (If you're not familiar with the basics of using HTTP with MIDP, or with the Generic Connection Framework, you should first read Qusay Mahmoud's article, which covers the necessary groundwork).

Using this reasoning, our first attempt at emulating the browser might look like this:

This code extract shows a class called Fetcher that has a method called fetch(), which requests information about a book whose ISBN is in an object of type BookInfo (which will be shown in the second article in this series), using the algorithm that was just described. The expectation is that, once the request has been sent, the HTML page for the book will be accessible from the input stream obtained from the HttpConnection's openInputStream() method. If this were a J2SE program and we had used a URL object to get a URLConnection and then made the same request as the one shown here, we would indeed get the HTML page from the input stream. Unfortunately, in the J2ME world, things are a little different.

We deliberately made this example more difficult by using a POST request instead of GET, in order to show you how different the MIDP HTTP implementation is from its J2SE counterpart when the server does not directly provide the data that you require. Instead of replying with a response code of 200 (or, more correctly, HttpConnection.HTTP_OK), the Amazon Web server sends a response code of 302, without any useful data. Since the data is missing, the usual, simple-minded approach of reading the content of the input stream isn't going to work here. So why is the server sending this response code, and what should we do about it?

Response code 302 is one of several codes that a Web server can use to indicate a redirection. The full list of these codes, and their official meanings, can be found in Table 1. A complete specification of the HTTP client's expected follow-up action when receiving each of these codes can be found in the HTTP 1.1 specification.

Table 1: HTTP redirect response codes

Code

Meaning

301

Moved permanently

302

Found

303

See other

305

Use proxy

307

Temporary redirect

These codes require the client to look for the requested resource at a different URL, which is included with the server's response in a header called Location. As well as connecting to a different location, if the server responsed with either 302 or 303, and the original request was a POST, the new request should instead be a GET; in all other cases, the original request method should be used.

There is no guarantee that a second request following a redirection will result in the required HTML page being returned, because multiple redirections are permitted. In other words, we have to keep following redirections until we get to the actual location of the information that we need. In order to avoid loops caused by incorrect server configuration, however, it is normal to impose an upper limit of the number of times a redirection will be followed, or to detect a loop by keeping a history of redirections.

In terms of our Fetcher class, the need to follow server redirections means that we convert the simple fetch method shown earlier into a loop that terminates when either the data is returned, an error occurs, or we get redirected too many times. Each pass of the loop will use the Connector class's open() method to open a new connection to the URL obtained from the previous redirection. The final version of this method is shown below, with the most important changes shown in bold.

As you can see, the first pass of the loop makes a POST request using the original URL, writing the query to the output stream obtained from the openOutputStream(). If a redirect is returned, the new URL is obtained from the Location header of the response and the request method is converted to GET if necessary, so that on the next pass the query will not be written. Conversion to a GET request implies that the server obtains all of the necessary information to locate the resource from the URL in the Location field.

In fact, in the case of the Amazon Web server, the URL that is supplied is exactly the one that you finally see in the Web browser when the page is displayed, and which contains the book's ISBN. Eventually, the server should return a response code of 200, at which point the HTML can be read from the input stream. This, however, is not the end of our problems, as we'll see in the next section.

Before moving on, you are probably wondering why it is that a J2SE application can get away without concerning itself with server redirects, whereas a MIDlet cannot. The answer to this question is very simple -- in J2SE, the code to handle redirects is built into the core libraries, and therefore happens automatically without the application being aware of it (although it can be turned off using the setFollowRedirects() and setInstanceFollowRedirects() method of java.net.HttpURLConnection). Unfortunately, the MIDP HTTP implementation does not include this feature.

A final word of caution -- if you think you can avoid the problems shown here by working out what the "real" URL is and using that to make the initial request, think again! In some cases, that might be possible, but in other cases it won't be. If you are faced with writing a J2ME client to interface with somebody else's server, the only way to work out what you have to do is try it and see -- or ask the server's owner, if that is possible. Be warned, though, that application servers hosting the server side of J2EE applications might use the redirection techniques shown here to point your client to a different URL following authentication, which is itself a topic that requires special treatment (similar to that shown here) for a MIDP application. That, however, is beyond the scope of this article.

How Much Data Will I Get?

Having connected to the server and successfully requested the HTML page for a book, we now need to read the response. The most obvious question to ask at this point is how much data are we going to receive or, more to the point, how can we tell when we have read all of the data that we're going to get? Answering this question is not as simple as you might think.

Back in the early days of the Web, each exchange between a browser and a Web server required a separate TCP/IP connection, which the browser would use to send the HTTP request headers and any accompanying data, then wait for server to reply. Having sent all of its reply, the server would close the connection. With this simple arrangement, the browser could simply read data until it detected that the connection had been closed, so there was no need for the server to explicitly indicate how much data it was going to send.

Unfortunately, this mode of operation turns out to be very expensive in most cases, because the time required to create a TCP/IP connection is often very significant when compared to the time spent subsequently transferring the data. For this reason, version 1.1 of the HTTP protocol allows the browser and the server to establish a TCP/IP connection and then use it more than once to exchange messages. In this mode, usually called "keepalive mode," the browser will typically send a request, read the server's response, send another request, and so on. Since TCP/IP is a stream-based rather than a message-based protocol, and since the connection is no longer being closed, there is no way for either the server or the browser to know when it has received all of the data from its peer, unless the data itself contains an embedded length. To solve this problem, HTTP 1.1 includes a Content-Length header, which can be used by the browser when sending a request and by the server in its response, to indicate how many bytes of data follow the HTTP headers.

The MIDP specification requires that support for HTTP 1.1 be provided and, therefore, keepalive mode is available (and, in fact, it is the default). A MIDP client application, therefore, appears to have two choices:

Use the HTTP 1.0 method with no re-use of the connection.

Use the HTTP 1.1 method, which might be more efficient for some applications.

To use the HTTP 1.0 method, you have to add an HTTP header called Connection with value Close. This requests that the server closes the connection after all of its data has been sent. To retrieve the data, you simply read until an end-of-file condition is reached.

To use the HTTP 1.1 method, you omit the Connection header and, if you are sending data along with the request, you also need to add the Content-Length header to specify the length of the data. In practice, though, the MIDP implementation includes this header for you.

In most cases, the likely result is that the server will return a response including a Content-Length header of its own, which you can retrieve using the HttpConnectiongetLength() method and then loop until you have read that many bytes from the input stream. However, even though you may ask the server to keep the connection open, it is not obliged to do so -- it may still close the connection after sending its reply and, if it decides to do this (or if it is, in fact, an HTTP 1.0 server that doesn't understand keepalives), it is not obliged to send you a Content-Length header. This means that even if you allow keepalives, you still have to be prepared to handle the alternative.

In order to keep the code for this article as simple as possible, our bookstore client opts for the HTTP 1.0 mode of operation by including the Connection header with value Close. However, if you expect to exchange more than one pair of messages with the same server in a reasonably short space of time, you should allow the server to keep the connection open, in which case you have to be prepared either to use the length in the Content-Length header, if the server returns it, or read until the connection is closed, if it does not, using code that looks something like this (where only the essential details are shown):

When the client and server operate in HTTP 1.1 keepalive mode, it is still possible that the getLength() method will return -1, even if the TCP/IP connection is not going to be closed to mark the end of the server's response. This can happen when the server chooses to use chunked mode when returning its response. Setting up a Content-Length header requires that the software (such as a servlet) that is responsible for generating the reply knows in advance how much data it is going to send. In some cases, this is not practical. Chunked mode allows the server to send out a response in chunks, where each individual chunk is preceded by its length and the end of the reply is marked by a chunk with length 0. The format of the body of a chunked HTTP reply message looks something like this:

19\r\n
This is a chunk of data. \r\n
16\r\n
This is another chunk.\r\n
0\r\n

The first line encodes the size of the following chunk in hexadecimal, followed by the characters \r\n. The data follows, on a single logical line, followed again by \r\n. The end of the reply is marked by a chunk of zero length, as shown.

The MIDP HTTP implementation takes care of managing chunks for you, so your code does not need to detect chunk boundaries and extract the length parts, etc. -- you just see a single stream of bytes which, in the example above would be:

This is a chunk of data. This is another chunk.

Since the complete length is not known until the last chunk has been read (and the HTTP 1.1 specification explicitly requires that a Content-Length header must not be included when using chunked encoding), calling getLength() when the server elects to supply a chunked response always results in -1 being returned, and you have no option but to read until the read() method returns -1.

Incidentally, the MIDP HTTP implementation may choose to write the body of your MIDlet's HTTP request to the server in chunked form. This is invisible to the code, but not to the server. Strictly speaking, this should not be a problem, because MIDP only supports interworking with HTTP 1.1 servers, which are required to accept chunked data. In the real world, however, it is sometimes necessary to communicate with an HTTP 1.0 server or application, which would not understand chunked encoding. You can avoid having your request sent in chunked mode by following both of the following rules:

Make sure that the message body is no more than 256 bytes long.

Do not call the flush() method on the output stream. Instead, write your message body to the output stream before opening the input stream or trying to access reply headers. This way, your message body will be sent without requiring a flush() call.

Obviously, this advice depends on the current MIDP HTTP implementation. Nevertheless, if you absolutely must communicate with a server that does not understand chunking, you have no choice but to rely on it.

Reading and Interpreting the Data

Now that we know how to detect when we have received all of the server's reply, the last thing we have to do is extract the information that we need from it, which is simply a matter of looking for certain strings in the HTML page. The book title, for example, follows the fixed string "buying info:", making up the text between the end of that string and the next end of line characters. Since the String class has convenient methods (such as indexOf) that make searching for substrings easy, the most obvious approach would be to read all of the server's reply into a single String, using code like this:

Notice that we use a ByteArrayOutputStream to gather the bytes that we receive and then call its toString() method to get the result in String form. This is convenient, not only because it manages the buffering of the data internally for the case where we don't know in advance exactly how much data we will receive, but also because the toString() method performs the necessary conversion from bytes to Unicode characters. Strictly speaking, this code should extract the character encoding of the received bytes from the HTTP headers of the server reply, but for simplicity we have assumed that the bytes are encoded using the client platform's default encoding. Since MIDP platforms are not guaranteed to support more than one encoding, this is an assumption you will normally have to make, in any case.

The problem with this simple approach is that while it would work on a J2SE platform, it requires too much memory for many of today's MIDP devices. Amazon's Web pages contain a lot of information -- in fact, a typical page would be at least 40KB long. Gathering this into a single String means that we have to be able to hold at least one copy of the whole page in memory (and the code shown above actually involves more than one copy). Unfortunately, this is not always possible. On a typical PalmOS device with 8MB of memory, the maximum memory available for the Java heap, which has to hold all of the objects and dynamic data for the Java VM, is only 64KB! The result of using this approach in such a resource-constrained environment is likely to be an OutOfMemoryError!
The memory limitation can be overcome by reading the page content in relatively small chunks and scanning for the strings we are looking for in each section as we read it. The details of this are a little messy and not particularly interesting, so we'll just take a brief look at some of the code to demonstrate how it works. If you want to see the complete implementation, you'll find it in the source files BookInfo.java and InputHelper.java.

To get the information that we require, we need to do the following:

Find the string "buying info:". This is followed by
the book title, which needs to be extracted and stored.

Find the string "Based on"" and then get the number of
reviews, which follows it.

Find the string "Sales Rank:", which is followed by the
book's sales ranking.

Because we are going to process the page content as we receive it and without storing much of it, it isn't possible to move backwards, so we have to search for these strings in the order shown. The major problem that we have to deal with is that we won't necessarily ever have any of these strings completely in memory at the same time -- we might find, for example, buying in one buffer full of data and info in the next, or we might receive each character individually. To keep the code readable, we create a class called InputHelper that hides the details of working with the characters as they arrive from the input stream and offers higher-level methods that provide the neccessary searching capability. The following code shows how this class is used to get the book title, its sales rank, and the number of reviews from the input stream, where the variable helper is an instance of the InputHelper class:

boolean found = helper.moveAfterString("buying info: ");
if (!found) {
return;
}
// Gather the title from the rest of this line
StringBuffer titleBuffer = helper.getRestOfLine();
// Look for the number of reviews
found = helper.moveAfterString("Based on ");
if (!found) {
return;
}
// Gather the number of reviews from the current location
String reviewString = helper.gatherNumber();
// Look for the sales rank
found = helper.moveAfterString("Sales Rank: ");
if (!found) {
return;
}
// Gather the number from the current location
String rankingString = helper.gatherNumber();

This code is, I think you'll agree, perfectly readable. The messy part is hidden in the InputHelper class. Here, for example, is how the gatherNumber() method, which scans forward until it finds a sequence of characters representing a number and collects them all into a string, is implemented:

The key to this method, and most of the others in this class, is the helper method getNext(), which returns a single character from the input stream. For efficiency, instead of calling the read() method of the InputStream, obtained from the HttpConnection each time getNext() is invoked, the data is held in a relatively small buffer which is filled when it is empty and returned from there. In this implementation, a 1024-byte buffer is used. The code shown here is written for the case in which the server will close the connection after sending the reply (or will use chunked encoding). To ensure that this will happen, the client includes a Connection: Close header with its request. Modifying the code to allow reuse of the connection if the server allows it is, however, fairly simple and might be undertaken as an exercise.

The gatherNumber() method calls getNext() whenever it needs a character, skipping until it finds a digit and then collecting digits until it finds something that is not numeric. The InputHelper class has several other methods used in processing the Web page returned by the server that work in exactly the same way. Writing the code in this way is slightly more complex than it would be if we could work with the entire reply in the form of a String and is almost certainly a little slower, but the environment in which the code executes leaves us with no real choice.

Summary

Although on the surface it appears that a MIDP HTTP client would be very much like one written for J2SE, this article has shown that, in the general case, you can't simply port existing J2SE code and expect it to work unchanged on a cell phone or PDA. The MIDP HTTP support currently presents a lower-level interface than its J2SE counterpart, requiring you to handle details, such as server redirection, than are taken care of automatically for desktop devices. Furthermore, the example shown in this article emphasized the ever-present need to be aware of the resource limitations imposed by MIDP devices and to restructure your code accordingly. In the second part of this article, we'll look at how to store the book's title, ISBN, and other information on a mobile device so that the user can keep track of changes without needing to enter the ISBN each time.

Kim Topley
has more than 25 years experience as a software developer and was one of the first people in the world to obtain the Sun Certified Java Developer qualification.