=head1 NAME

head-r E<ndash> Recurse through Web pages and issue HEAD requests

=head1 ABSTRACT

=begin comment

                            WARNING!

This .pod documentation was AUTOGENERATED from XHTML source!

Please be sure to edit the respective source document.  The
changes made here are likely to be LOST.

=end comment

=encoding UTF-8

Head-r is a free Perl program that recursively follows links located at (HTML) Web pages hosted on an HTTP server, and performs C<HEAD> upon links of interest to the user.

The intended use for this program is to create URI lists for later selective mirroring of file-hosting sites.



=head1 SYNOPSIS


    head-r [-v|--verbose] [-j|--bzip2|-z|--gzip]
        [--include-re=RE] [--exclude-re=RE]
        [--depth=N] [--info-re=RE] [--descend-re=RE]
        [-i|--input=FILE]... [-o|--output=FILE]
        [-P|--no-proxy] [-U|--user-agent=USER-AGENT]
        [-w|--wait=DELAY]
        [--] [URI]...


=head1 BASIC USAGE


Arguably, the most important Head-r options are C<--info-re=> and C<--descend-re=>, which determine (by means of regular expressions) which URIs will be considered for mere C<HEAD> requests, and which ones Head-r will try to get more URIs from.


=head2 Simplistic, no-recursion example


For the following example, we’ll use C<.> – a regular expression that matches any non-empty string – to allow Head-r to make C<HEAD> requests to both of the URIs given.


    $ head-r --info-re=. \
          -- http://example.org/ http://example.net/ 
    http://example.org/	1381334900	1	1270	200
    http://example.net/	1381334903	1	1270	200


The fields are delimited with ASCII HT (also known as TAB) codes, and are as follows:


=over

=item 1. 

URI;

=item 2. 

timestamp (in seconds since system-dependent epoch; see also Unix time);

=item 3. 

recursion depth used when considering this URI;

=item 4. 

the length of the response in octets (as per the Content-Length: HTTP reply header);

=item 5. 

HTTP status code of the reply.

=back


=head2 Recurse once example


For the following example, we’ll also enable actual recursion (still at I<maximum depth> of 1), by using the C<--descend-re=/\$> option.


    $ head-r --info-re=. --descend-re=/\$ \
          -- http://example.org/ http://example.net/ 
    http://example.org/	1381337824	1	1270	200
    http://www.iana.org/domains/example	1381337829	0	200
    http://example.net/	1381337830	1	1270	200


As could be seen, at L<http://example.org/> Head-r found another URI to consider: L<http://www.iana.org/domains/example>, which it followed and issued a C<HEAD> request for.

It’s easy to check that L<http://example.net/> actually also references the same URI. However, as Head-r I<remembers the URIs it processes> (along with the recursion depth at the point) no other request was issued.


=head2 Limiting HEAD requests


Consider now that the resource we’re to recurse through references URIs that are out of our interest. For the following example, we’ll use a more selective regular expression than C<.> we’ve used above.


    $ head-r --{info,descend}-re=wikipedia\\.org/wiki/ \
          -- http://en.wikipedia.org/wiki/Main_Page 
    http://en.wikipedia.org/wiki/Main_Page	1381339589	1	61499	200
    . . .
    http://en.wikipedia.org/w/api.php?action=rsd
    http://creativecommons.org/licenses/by-sa/3.0/
    . . .
    http://meta.wikimedia.org/
    http://en.wikipedia.org/wiki/Wikipedia	1381339589	0	609859	200
    http://en.wikipedia.org/wiki/Free_content	1381339589	0	124407	200
    . . .


(Please note that we’ve just used the Bash C<{,}> expansion to pass the same regular expression to both C<--info-re=> and C<--descend-re=>. Be sure to adjust to the command line interpreter actually in use.)

In the output above, a number of URIs came without any of the usual information. These URIs were found by Head-r, but as they matched neither “info” (C<--info-re=>) nor “descend” (C<--descend-re=>) regular expressions specified, no action was done to them. The URIs are still output, however, just in case we may decide to adjust the regular expressions themselves.


=head2 Skipping unwanted URIs altogether


The C<--include-re=> and C<--exclude-re=> regular expressions are considered before all the other ones, and I<currently> have the following semantics:


=over

=item 1. 

the inclusion regular expression is applied first; the URI will be considered if it matches one;

=item 2. 

unless decided at the step above, the exclusion regular expression is then applied; the URI will not be considered if it matches one;

=item 3. 

unless decided by the rules above, the URI will be considered.

=back


If none of these options are given, any URI will be considered by Head-r.

The following example exploits these options to further limit the output of Head-r for the case above.


    $ head-r --{include,descend}-re=wikipedia\\.org/wiki/ \
          --{info,exclude}-re=. \
          -- http://en.wikipedia.org/wiki/Main_Page 
    http://en.wikipedia.org/wiki/Main_Page	1381341336	1	61499	200
    http://en.wikipedia.org/wiki/Wikipedia	1381341337	0	609859	200
    http://en.wikipedia.org/wiki/Free_content	1381341337	0	124407	200
    http://en.wikipedia.org/wiki/Encyclopedia	1381341337	0	151164	200
    http://en.wikipedia.org/wiki/Wikipedia:Introduction	1381341337	0	50687	200
    . . .


=head1 SAVING STATE BETWEEN SESSIONS


Head-r is capable of reading its own output, so to avoid issuing duplicate C<HEAD> requests, and also to discover the URIs of the resources to recurse into.


=head2 Restoring what was saved


Let us revisit one of our previous examples, which we’ll now alter to only issue a C<HEAD> request to a couple of pages:


    $ head-r --output=state.a \
          --info-re='/(Free_content|Wikipedia)$' \
          --descend-re=wikipedia\\.org/wiki/ \
          -- http://en.wikipedia.org/wiki/Main_Page 
    $ grep -E \\s < state.a 
    http://en.wikipedia.org/wiki/Main_Page	1381417546	1	61499	200
    http://en.wikipedia.org/wiki/Wikipedia	1381417546	0	609859	200
    http://en.wikipedia.org/wiki/Free_content	1381417546	0	124407	200
    $ 


Now, why not to include a few more pages, such as all the pages with the names starting with C<F>?


    $ head-r \
          --input=state.a --output=state.b \
          --info-re=/wiki/F \
          --descend-re=wikipedia\\.org/wiki/ 
    $ grep -E \\s < state.b 
    http://en.wikipedia.org/wiki/File:Diary_of_a_Nobody_first.jpg	1381417906	0	34344	200
    http://en.wikipedia.org/wiki/File:Progradungula_otwayensis_cropped.png	1381417906	0	30604	200
    http://en.wikipedia.org/wiki/File:AW_TW_PS.jpg	1381417907	0	33297	200
    http://en.wikipedia.org/wiki/Fran%C3%A7ois_Englert	1381417907	0	87860	200
    http://en.wikipedia.org/wiki/File:Washington_Monument_Dusk_Jan_2006.jpg	1381417907	0	83137	200
    http://en.wikipedia.org/wiki/File:Walt_Disney_Concert_Hall,_LA,_CA,_jjron_22.03.2012.jpg	1381417907	0	67225	200
    http://en.wikipedia.org/wiki/Frank_Gehry	1381417907	0	152838	200
    $ 


Note that while our C<--info-re=> has obviously covered L<http://en.wikipedia.org/wiki/Free_content>, no C<HEAD> request was made to the page, as our C<--input=state.a> file already had the relevant information.

Also, as all the URIs we wanted for Head-r to consider were already listed in C<state.a>, it was unnecessary to specify any URIs at the command line. When the URIs come from I<both> command line arguments I<and> C<--input=> files, those coming from command line I<are considered first.>


=head2 Compression


As recursing through large Web sites may result in large output lists, Head-r provides support for compression of output data.

The C<--bzip2> (C<-j>) and C<--gzip> (C<-z>) options select the compression method to use for the output file (either specified with C<--output=>, or standard output.) Head-r, however, will exit with an error if compression is enabled and the output goes to a terminal device.

Head-r transparently decompresses the files given as inputs (C<--input=>), thanks to the L<IO::Uncompress::AnyUncompress|IO::Uncompress::AnyUncompress> library.


=head1 ADJUSTING HTTP CLIENT BEHAVIOR


There’re two options which influence the behavior of the HTTP client used by Head-r: C<--wait=> (C<-w>) and C<--user-agent=> (C<-U>.)

The C<--wait=> option specifies the amount of time, in seconds, to wait between two consecutive HTTP requests. The default is about 2.7 seconds.

The C<--user-agent=> option specifies the value for the C<User-Agent:> header to use in HTTP requests, and may come handy should the target server block access based on this header’s data. The default is composed of the string C<HEAD-R-Bot/>, the Head-r’s own version, and the identity of the libwww-perl library used. For example: C<HEAD-R-Bot/0.1 libwww-perl/6.05>.


=head1 BUGS


Please consider reporting any bugs in the Head-r software not listed below via the CPAN RT, L<https://rt.cpan.org/Public/Dist/Display.html?Name=head-r>. The bugs in this documentation should be reported to the respective Wikibooks Talk page – or you may actually fix them yourself!

As for any other automatic retrieval tool, it isn’t impossible to abuse Head-r to cause excessive load on third-party servers. The user is advised to consider the network environment when using the tool, and especially when lowering the C<--wait=> setting, and raising the maximum recursion C<--depth=> beyond reasonable values.

There’s currently no way to disable the C</robots.txt> file processing.

The code only tries to retrieve URIs from content marked with C<text/html> media type, even though it seems as if the support for C<application/xhtml+xml> (and perhaps several other XML-based types, such as SVG) could be implemented rather easily.

The resource to retrieve URIs from is first loaded into memory, while it should be possible to process it on-the-fly.

The handling of recursion depths retrieved from C<--input=> files may be somewhat unintuitive, and out of the user’s control. (Although it’s still possible to edit such files using third-party tools, such as AWK.)

The code implements a trivial work-around for the long-standing L<Net::HTTP|Net::HTTP> bug #29468.


=head1 AVAILABILITY


The code could be downloaded from a Git repository, like:


    $ git clone -- \
          http://am-1.org/~ivan/archives/git/head-r-2013.git/ head-r 


A Gitweb interface is available from L<http://am-1.org/~ivan/archives/git/gitweb.cgi?p=head-r-2013.git>.


=head1 AUTHOR


Head-r is written by Ivan Shmakov.

Head-r is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This documentation is a free collaborative project going on at Wikibooks, and is available under the Creative Commons Attribution/Share-Alike License (CC BY-SA) version 3.0.
