Man page of lwptut

lwptut

NAME

lwptut -- An LWP Tutorial

DESCRIPTION

LWP (short for ``Library for WWW in Perl'') is a very popular group of Perl modules for accessing data on the Web. Like most Perl module-distributions, each of LWP's component modules comes with documentation that is a complete reference to its interface. However, there are so many modules in LWP that it's hard to know where to start looking for information on how to do even the simplest most common things.

Really introducing you to using LWP would require a whole book --- a book that just happens to exist, called Perl &LWP. But this article should give you a taste of how you can go about some common tasks with LWP.

Getting documents with LWP::Simple

If you just want to get what's at a particular URL, the simplest way to do it is LWP::Simple's functions.

In a Perl program, you can call its

&quot;get($url)&quot;

function. It will try getting that URL's content. If it works, then it'll return the content; but if there's some error, it'll return undef.

There are other useful functions in LWP::Simple, including one function for running a HEAD request on a URL (useful for checking links, or getting the last-revised time of a URL), and two functions for saving/mirroring a URL to a local file. See the LWP::Simple documentation for the full details, or chapter 2 of Perl &LWP for more examples.

The Basics of the LWP Class Model

LWP::Simple's functions are handy for simple cases, but its functions don't support cookies or authorization, don't support setting header lines in the HTTP request, generally don't support reading header lines in the HTTP response (notably the full HTTP error message, in case of an error). To get at all those features, you'll have to use the full LWP class model.

While LWP consists of dozens of classes, the main two that you have to understand are LWP::UserAgent and HTTP::Response. LWP::UserAgent is a class for ``virtual browsers'' which you use for performing requests, and HTTP::Response is a class for the responses (or error messages) that you get back from those requests.

The basic idiom is

&quot;$response = $browser-&gt;get($url)&quot;

, or more fully illustrated:

# Early in your program:

use LWP 5.64; # Loads all important LWP classes, and makes
# sure your version is reasonably recent.

object, which is of class HTTP::Response. You really need only one browser object per program; but every time you make a request, you get back a new HTTP::Response object, which will have some interesting attributes:

•

A status code indicating success or failure (which you can test with

&quot;$response-&gt;is_success&quot;

).:

•

An HTTP status line that is hopefully informative if there's failure (which you can see with

&quot;$response-&gt;status_line&quot;

, returning something like ``404 Not Found'').:

•

A MIME content-type like ``text/html, ``image/gif, ``application/xml'', etc., which you can see with

&quot;$response-&gt;content_type&quot;

:

•

The actual content of the response, in

&quot;$response-&gt;decoded_content&quot;

. If the response is HTML, that's where the HTML source will be; if it's a GIF, then

&quot;$response-&gt;decoded_content&quot;

will be the binary GIF data.:

•

And dozens of other convenient and more specific methods that are documented in the docs for HTTP::Response, and its superclasses HTTP::Message and HTTP::Headers.:

Adding Other HTTP Request Headers

The most commonly used syntax for requests is

&quot;$response = $browser-&gt;get($url)&quot;

, but in truth, you can add extra HTTP header lines to the request by adding a list of key-value pairs after the URL, like so:

If you were only ever changing the 'User-Agent' line, you could just change the

$browser

object's default line from ``libwww-perl/5.65'' (or the like) to whatever you like, using the LWP::UserAgent

&quot;agent&quot;

method:

$browser-&gt;agent('Mozilla/4.76 [en] (Win98; U)');

Enabling Cookies

A default LWP::UserAgent object acts like a browser with its cookies support turned off. There are various ways of turning it on, by setting its

&quot;cookie_jar&quot;

attribute. A ``cookie jar is an object representing a little database of all the HTTP cookies that a browser can know about. It can correspond to a file on disk (the way Netscape uses its cookies.txt'' file), or it can be just an in-memory object that starts out empty, and whose collection of cookies will disappear once the program is finished running.

To give a browser an in-memory empty cookie jar, you set its

&quot;cookie_jar&quot;

attribute like so:

$browser-&gt;cookie_jar({});

To give it a copy that will be read from a file on disk, and will be saved to it when the program is finished running, set the

Of course, using a regexp to match hrefs is a bit simplistic, and for more robust programs, you'll probably want to use an HTML-parsing module like HTML::LinkExtor or HTML::TokeParser or even maybe HTML::TreeBuilder.

Other Browser Attributes

LWP::UserAgent objects have many attributes for controlling how they work. Here are a few notable ones:

•

&quot;$browser-&gt;timeout(15);&quot;

This sets this browser object to give up on requests that don't answer within 15 seconds.

:

•

&quot;$browser-&gt;protocols_allowed( [ 'http', 'gopher'] );&quot;

This sets this browser object to not speak any protocols other than HTTP and gopher. If it tries accessing any other kind of URL (like an ``ftp: or ``mailto: or ``news: URL), then it won't actually try connecting, but instead will immediately return an error code 500, with a message like ``Access to 'ftp' URIs has been disabled.

This changes how the browser object will identify itself in the default ``User-Agent line is its HTTP requests. By default, it'll send "libwww-perl/versionnumber``, like libwww-perl/5.65". You can change that to something more descriptive like this:

) to avoid making too many requests too often. How long it will pause for, is by default one minute --- but you can control it with the

&quot;$browser-&gt;delay(

minutes

)&quot;

attribute.

For example, this code:

$browser-&gt;delay( 7/60 );

...means that this browser will pause when it needs to avoid talking to any given server more than once every 7 seconds.

:

For more options and information, see the full documentation for LWP::RobotUA.

Using Proxies

In some cases, you will want to (or will have to) use proxies for accessing certain sites and/or using certain protocols. This is most commonly the case when your LWP program is running (or could be running) on a machine that is behind a firewall.

To make a browser object use proxies that are defined in the usual environment variables (

&quot;HTTP_PROXY&quot;

, etc.), just call the

&quot;env_proxy&quot;

on a user-agent object before you go making any requests on it. Specifically:

use LWP::UserAgent;
my $browser = LWP::UserAgent-&gt;new;
# And before you go making any requests:
$browser-&gt;env_proxy;

For more information on proxy parameters, see the LWP::UserAgent documentation, specifically the

&quot;proxy&quot;

,

&quot;env_proxy&quot;

, and

&quot;no_proxy&quot;

methods.

HTTP Authentication

Many web sites restrict access to documents by using ``HTTP Authentication. This isn't just any form of ``enter your password restriction, but is a specific mechanism where the HTTP server sends the browser an HTTP code that says ``That document is part of a protected 'realm', and you can access it only if you re-request it and add some special authorization headers to your request''.

For example, the Unicode.org admins stop email-harvesting bots from harvesting the contents of their mailing list archives, by protecting them with HTTP Authentication, and then publicly stating the username and password (at

If your LWP installation doesn't have HTTPS support set up, then the response will be unsuccessful, and you'll get this error message:

Error at https://www.paypal.com/
501 Protocol scheme 'https' is not supported
Aborting at paypal.pl line 7. [or whatever program and line]

@]

If your LWP installation does have HTTPS support installed, then the response should be successful, and you should be able to consult

$response

just like with any normal HTTP response.

For information about installing HTTPS support for your LWP installation, see the helpful README.SSL file that comes in the libwww-perl distribution.

Getting Large Documents

When you're requesting a large (or at least potentially large) document, a problem with the normal way of using the request methods (like

&quot;$response = $browser-&gt;get($url)&quot;

) is that the response object in memory will have to hold the whole document --- in memory. If the response is a thirty megabyte file, this is likely to be quite an imposition on this process's memory usage.

A notable alternative is to have LWP save the content to a file on disk, instead of saving it up in memory. This is the syntax to use: