@node LWP/RobotUA, LWP/Simple, LWP/Protocol, Module List
@unnumbered A class for Web Robots


@unnumberedsec NAME

LWP::RobotUA - A class for Web Robots

@unnumberedsec SYNOPSIS

@example
require LWP::RobotUA;
$ua = new LWP::RobotUA @'my-robot/0.1@', @'me@@foo.com@';
$ua->delay(10);  # be very nice, go slowly
...
# just use it just like a normal LWP::UserAgent
$res = $ua->request($req);
@end example

@unnumberedsec DESCRIPTION

This class implements a user agent that is suitable for robot
applications.  Robots should be nice to the servers they visit.  They
should consult the @file{robots.txt} file to ensure that they are welcomed
and they should not send too frequent requests.

But, before you consider writing a robot take a look at
<URL:http://info.webcrawler.com/mak/projects/robots/robots.html>.

When you use a @emph{LWP::RobotUA} as your user agent, then you do not
really have to think about these things yourself.  Just send requests
as you do when you are using a normal @emph{LWP::UserAgent} and this
special agent will make sure you are nice.

@unnumberedsec METHODS

The LWP::RobotUA is a sub-class of LWP::UserAgent and implements the
same methods.  The use_alarm() method also desides whether we will
wait if a request is tried too early (if true), or will return an error
response (if false).

In addition these methods are provided:

@unnumberedsubsec $ua = LWP::RobotUA->new($agent_name, $from, [$rules])

Your robot@'s name and the mail address of the human responsible for
the robot (i.e. you) is required by the constructor.

Optionally it allows you to specify the @emph{WWW::RobotRules} object to
use.

@unnumberedsubsec $ua->delay([$minutes])

Set the minimum delay between requests to the same server.  The
default is 1 minute.

@unnumberedsubsec $ua->rules([$rules])

Set/get which @emph{WWW::RobotRules} object to use. 

@unnumberedsubsec $ua->no_visits($netloc)

Returns the number of documents fetched from this server host.

@unnumberedsubsec $ua->host_wait($netloc)

Returns the number of seconds you must wait before you can make a new
request to this host.

@unnumberedsubsec $ua->as_string

Returns a text that describe the state of the UA.
Mainly useful for debugging.

@unnumberedsec SEE ALSO

@xref{LWP/UserAgent,LWP/UserAgent},, @xref{WWW/RobotRules,WWW/RobotRules},

@unnumberedsec AUTHOR

Gisle Aas <aas@@sn.no>

