Extending Web Services Using Other Web Services

How to create a useful new Web service by tapping in to the power of two other freely available Web services.

Last month, we looked at the latest incarnation of Web services
offered by on-line giant Amazon. Amazon was one of the first companies
to embrace Web services, and although some of its newer offerings
require payments on a monthly or per-query basis, basic catalog
searches are still available free of charge.

If we think of each individual Web service as a function call, we
can think of a collection of Web services, such as Amazon Web Services
(AWS), as a software
library. And although we can certainly create interesting applications
with such libraries, it is often useful to create new libraries that
sit on top of the existing ones. In many ways, the history of
software is the history of creating increasingly powerful abstractions
by stacking libraries on top of one another. Outside of the
classroom, most of us haven't ever had to implement a sort algorithm
or create a buffered I/O library, simply because such things have been
written and optimized by previous generations of programmers.

I thus believe that it's useful for us to consider AWS not as a set of
routines that we can incorporate into end-user programs, but rather
as a set of low-level libraries on top of which we can (and should)
create new libraries appropriate for our specific needs.

This month, we look at a simple example of what I mean. The
project will reflect my love of books. The Internet has made it
difficult for me to stop buying used books, because so many are
available at low prices. But, I'm fortunate to be spending several
years in Skokie, Illinois, which has an excellent public library.
Skokie's library has not only an extensive collection, but it also has a
Web-based interface to the book catalog. Our project for this month,
thus, is to create a Web service that integrates Amazon's catalog
with the information from the Skokie public library. In other words,
we're going to write a Web service that itself relies upon another Web
service. The input to our service will be an International Standard Book Number (ISBN); the output will be
an indication of the book's availability and price at Amazon and the
Skokie library.

In some ways, this Web service will duplicate the excellent Book
Burro plugin for the Firefox Web browser, which I often use to find
the best bargains. And indeed, Book Burro looks at both bookstores
and public libraries in order to find books. I recommend Book Burro
to everyone who uses Firefox. But, I believe that building your own
simple Web service, even if it duplicates the functionality of another
program, is a worthwhile endeavor.

Moreover, Web services have the advantage of being available from any
programming language and any application. I can implement my Web
service using Ruby, and people will still be able to access it from
Java, Python, Perl or virtually any other language. In many ways,
this achieves what object broker middleware services like CORBA had
promised, only without the baggage that made CORBA a more complex (but
arguably more secure and rich) programming platform. It makes a Web
service more powerful than a simple software library, because it can be accessed from any platform or language, so long as the
requesting computer is connected to the Internet.

Searching the Catalog

In order to integrate an ISBN search for the Skokie library, we're
going to need a way to query the library for information about
book availability. Unfortunately, my library doesn't have a Web
services API for querying its database. But, it does have the
next-best thing, namely a simple Web interface that we can query.

There are several ways to look through the output from a Web page.
Because many sites now use HTML that can be parsed as if it were XML,
we might want to use an XML-parsing library to read through the
response from the library's Web site, looking for particular text in
specific places.

Much as I might like the idea of such an approach, I'm probably not
the only Web developer who takes a more practical, quick-and-dirty
look. I have used my library's Web site enough times to know that
there is a limited number of responses it might send back to me.
As a result, I'll use the reliable, if somewhat stupid, approach of
looking for particular cues in the HTTP response.

Our program (skokie-lookup.rb, Listing 1) is written in Ruby, a
language I have grown to enjoy more and more over the past few
months. We begin by importing the included Net::HTTP module, which
defines classes and methods that provide HTTP-based communication.

We then check to make sure that we have at least one command-line
argument, by looking at the built-in ARGV array. If the length of
ARGV is 0, we know we weren't passed any arguments, and
we should give the user a brief indication of how the program should
be used.

Then, we set up a number of variables that will be needed later on.
The output variable is a string to which we will add any output we
need to send to the user. We also create three Regexp (regular
expression) objects, which we will use in our loop.

Next comes the meat of the program. We iterate over each element of
ARGV, first checking that it is a ten-character ISBN containing only
numbers and the letter X. We then query the Skokie library's Web site
for that ISBN, passing Net::HTTP.get_response the hostname and path to
the program we want. The HTTP response, including its headers and
body, is then available in our response variable.

Now we compare the response body against our three regular
expressions, checking which it matches. Using Ruby's << operator for
concatenation, we add an appropriate message to the output variable
for each ISBN. Finally, just before the program exits, it gives a
full report of ISBNs.