# $File: //member/autrijus/PAR/lib/PAR/Intro.pod $
# $Revision: #8 $ $Change: 7518 $ $DateTime: 2003/08/14 07:50:47 $ vim: expandtab shiftwidth=4

=head1 NAME

PAR::Intro - Introduction to Perl Archive Toolkit

=head1 SYNOPSIS

# This is a presentation, not a module.

Note that a more extensive tutorial is now available online
as L<http://aut.dyndns.org/par-tutorial/> and has superceded
materials in this introduction.

=head1 DESCRIPTION

=head2 What is PAR (Perl Archive Toolkit)?

=over 4

=item *

Do what JAR (Java Archive) does for Perl

=over 4

=item *

Platform-independent, compressed file format (zip)

=item *

Aggregates modules, scripts and other files into one file

=item *

Easy to generate, update and extract

=back


=item *

Benefits of using PAR:

=over 4

=item *

Reduced download and deployment time

=item *

Saves disk space by compression and selective packaging

=item *

Version consistency: solves forward-compatibility problems

=item *

Community support: C<par@perl.org>

=back

=item *

You can also turn a PAR file into a self-contained script

=over 4

=item *

Bundles all necessary 3rd-party modules with it

=item *

Requires only core Perl to run on the target machine

=item *

If you use C<pp> to compile the script...

=item *

...you get an executable not even needing core perl

=back

=back

=head2 Getting Started

=over 4

=item *

First, generate a PAR file with modules in it:

    % zip foo.par Hello.pm
    % zip -r foo.par lib/       # grab all modules in lib/

=item *

Using modules stored inside a PAR file:

    % perl -MPAR=./foo.par -MHello
    % perl -MPAR=./foo -MHello  # the .par part is optional

=item *

Or put it in @INC and use it just like a directory:

    % perl -MPAR -Ifoo.par -MHello
    % perl -MPAR -Ifoo -MHello  # ditto

=back

=head2 Command-line Tools

=over 4

=item *

Use C<pp> to scan scripts and store dependencies as a PAR file:

    % pp -p source.pl           # makes 'source.par'
    % pp -B -p source.pl        # bundles core modules too

=item *

Use C<par.pl> to run files from a Perl Archive:

    % par.pl foo.par            # looks for 'main.pl' by default
    % par.pl foo.par test.pl    # runs script/test.pl in foo.par

=item *

Use C<parl> or C<parl.exe> to run files from a Perl Archive:

    % parl foo.par
    % parl foo.par test.pl

=back

=head2 Making Binary Executables

=over 4

=item *

The C<pp> utility can also generate binary executables:

    % pp -o packed.exe source.pl    # self-contained .exe
    % packed.exe                    # runs anywhere with the same OS

=item *

You can also bundle additional modules:

    # packs CGI + its dependencies, too
    % pp -o packed.exe -M CGI source.pl

=item *

Or pack one-liners:

    # turns one-liner into executable
    % pp -o packed.exe -e 'print "Hi!"'

=item *

Some notes:

=over 4

=item *

The command-line options of C<pp> are almost identical to C<perlcc>'s

=item *

Modules are read directly from the PAR file, not extracted

=item *

Shared object files (aka dll) are extracted with File::Temp

=item *

Tested on Win32, FreeBSD, Linux, AIX, Solaris, Darwin and Cygwin.

=back

=back


=head2 The Anatomy of a PAR file

=over 4

=item *

Modules can reside in different directories in a PAR file:

    /lib/                       # standard location
    /arch/                      # for creating from blib/ 
    /i386-freebsd/              # i.e. $Config{archname}
    /5.8.0/                     # i.e. Perl version number
    /5.8.0/i386-freebsd/        # combination of the two above
    /                           # casual packaging only

=item *

Scripts are stored in one of the two locations:

    /script/                    # standard location
    /                           # casual packaging only

=item *

Shared libraries may be architecture- or perl-version-specific:
 
    /shlib/(5.8.0/)?(i386-freebsd/)?

=item *

PAR files may recursively contain other PAR files:
 
    /par/(5.8.0/)?(i386-freebsd/)?
 
=item *

Special files:

    /MANIFEST                   # index of the PAR's contents
    /SIGNATURE                  # digital signature(s)
    /META.yml                   # dependency, license info, etc.
    /Build.PL                   # self-contained installer

=item *

Programs can use C<PAR::read_file($filename)> to read file contents inside PAR

=item *

Programs can use C<PAR::reload_libs()> to reload modules within changed PARs

=back

=head2 Derived Modules

=over 4

=item *

Apache::PAR

=over 4

=item *

Nathan Byrd's attempt to make self-contained Perl Handlers

=item *

Same as the WAR files for Java Servlets

=item *

Includes PerlRun and Registry handlers

=back

=item *

App::Packer::Backend::PAR

=over 4

=item *

Support module of Mattia Barbon's App::Packer suite

=item *

Makes it easy to pick-and-choose dependency scanners and packers

=item *

Fine-tuned distribution and packaging controls

=back

=item *

CPANPLUS::Dist::PAR

=over 4

=item *

Cross-platform PPM: Auto-generate PAR out of CPAN distributions

=item *

Use the bundled Build.PL to install PAR modules into system

=back

=back


=head2 Apache::PAR Demo

=over 4

=item *

In C<httpd.conf>:

    <VirtualHost *>
        <IfDefine MODPERL2>
        PerlModule Apache::ServerUtil
        </IfDefine>
        PerlModule Apache::PAR
        PARDir /opt/myapp
        PARFile /opt/myapp/myapp.par
    </VirtualHost>

=item *

In C<web.conf> inside C<myapp.par>:

    Alias /myapp/static/ ##PARFILE##/
    <Location /myapp/static>
        SetHandler perl-script
        PerlHandler Apache::PAR::Static
        PerlAddVar PARStaticDirectoryIndex index.html
        PerlSetVar PARStaticDefaultMIME text/html
    </Location>

    Alias /myapp/cgi-perl/ ##PARFILE##/
    <Location /myapp/cgi-perl>
        Options +ExecCGI
        SetHandler perl-script
        PerlHandler Apache::PAR::Registry
    </Location>

=back

=head2 Future Development

=over 4

=item *

Polish C<pp>'s features

=over 4

=item *

Handles corner dependency cases for LWP, Tk, DBI...

=item *

Optional encryption support (but *not* obscuring)

=item *

Become a worthy competitor to PerlApp and Perl2Exe

=back

=item *

Learning from JAR

=over 4

=item *

Making par.pl's command line interface in sync with jar's

=item *

Digital signatures for PAR packages using Module::Signature

=item *

File layout compatibility?

=back

=item *

Learning from FreeBSD Bento

=over 4

=item *

Smoke test and make PAR automatically for each CPAN upload

=item *

Provide binary packages for users without a compiler

=back

=back


=head2 Overview of PAR.pm's Implementation

=over 4

=item *

Here begins the scary part

=over 4

=item *

Grues, Dragons and Jabberwocks abound...

=item *

You are going to learn unpleasant things about Perl internals

=item *

Go home now if you have heart condition or digest problems

=back

=item *

PAR invokes five areas of Perl arcana:

=over 4

=item *

@INC code references

=item *

On-the-fly source filtering

=item *

Faking <DATA> filehandle with PerlIO::scalar and IO::Scalar

=item *

Overriding DynaLoader::bootstrap to handle XS modules

=item *

Making self-bootstrapping binary executables

=back

=item *

The first two only works on 5.6 or later

=over 4

=item *

PerlIO::scalar is 5.8-specific; IO::scalar only needs 5.005

=item *

DynaLoader and %INC are there since Perl 5 was born

=item *

PAR currently needs 5.6, but a 5.005 port is possible

=back

=back


=head2 Code References in @INC

=over 4

=item *

On 1999-07-19, Ken Fox submitted a patch to P5P

=over 4

=item *

To "enable using remote modules" by putting hooks in @INC

=item *

It's accepted to come in Perl 5.6, but only get documented by 5.8

=item *

Type 'perldoc -f require' to read the nitty-gritty details

=back

=item *

Code references in @INC may return a filehandle, or undef to 'pass':

    push @INC, \&my_sub;
    sub my_sub {
        my ($coderef, $filename) = @_;  # $coderef is \&my_sub
        open my $fh, "wget http://example.com/$filename |";
        return $fh;     # using remote modules, indeed!
    }

=item *

Perl 5.8 let you open a file handle to a string, so we just use that:

    open my $fh, '<', \($zip->memberNamed($filename)->contents);
    return $fh;

=item *

But Perl 5.6 does not have that, and I don't want to use temp files...

=back

=head2 Source Filtering without Filter::* Modules

=over 4

=item *

... Undocumented features to the rescue!

=over 4

=item *

It turns out that @INC hooks can return *two* values

=item *

The first is still the file handle

=item *

The second is a code reference for line-by-line source filtering!

=back

=item *

This is how C<Acme::use::strict::with::pride> works:

    # Force all modules used to use strict and warnings
    open my $fh, "<", $filename or return;
    my @lines = ("use strict; use warnings;\n", "#line 1 \"$full\"\n");
    return ($fh, sub {
        return 0 unless @lines; 
        push @lines, $_; $_ = shift @lines; return length $_;
    });

=item *

But we don't really have a filehandle for anything

=over 4

=item *

Another undocumented feature to the rescue

=item *

We can actually omit the first return value altogether:

    # Return all contents line-by-line from the file inside PAR
    my @lines = split /(?<=\n)/, $zip->memberNamed($filename)->contents;
    return (sub { $_ = shift(@lines); return length $_ });

=back

=back

=head2 Faking the <DATA> Handle

=over 4

=item *

The @INC filter stops when it sees C<__END__> or C<__DATA__>

=over 4

=item *

All contents below are lost

=item *

Breaks modules that read from the <DATA> filehandle

=item *

The same problem appears when we C<eval> the main.pl script

=back

=item *

Therefore, we insert a line before the final token to fake *DATA

=over 4

=item *

It has to be the final line to belong to the correct package

=item *

It has to happen in compile time but not inside a BEGIN block

=item *

Here is what I came up with (but no longer needed in recent versions):

    $DATACache{$file} = $1 if ($program =~ s/^__DATA__\n?(.*)//ms);
    if (eval {require PerlIO::scalar; 1}) {
        "use PerlIO::scalar".
        "  ( open(*DATA, '<:scalar', \\\$PAR::DATACache{'$key'}) ? () : () )";
    }
    elsif (eval {require IO::Scalar; 1}) {
        # This will first load IO::Scalar, *then* tie the handles.
        "use IO::Scalar".
        "  ( tie(*DATA, 'IO::Scalar', \\\$PAR::DATACache{'$key'}) ? () : () )";
    }
    else {
        # only dies when it's used
        "use PAR (tie(*DATA, 'PAR::_data') ? () : ())\n";
    }
    sub PAR::_data::TIEHANDLE { return bless({}, shift) }
    sub PAR::_data::AUTOLOAD { die "Please install IO::Scalar first!\n" }

=back

=back


=head2 Overriding DynaLoader::bootstrap

=over 4

=item *

XS modules have dynamically loaded libraries (C<.so> or C<.dll>)

=over 4

=item *

They cannot be loaded as part of a zip file, so we extract them out

=item *

But I don't want to make any temporary C<auto/> directories

=item *

So we have to intercept DynaLoader's library-finding process

=back

=item *

Module names are passed to C<bootstrap> for XS loading

=over 4

=item *

During the process, it calls C<dl_findfile> to locate the file

=item *

So we wrap around both functions:

    no strict 'refs'; no warnings 'redefine';
    $bootstrap   = \&DynaLoader::bootstrap;
    $dl_findfile = \&DynaLoader::dl_findfile;
    *{'DynaLoader::bootstrap'}   = \&_bootstrap;
    *{'DynaLoader::dl_findfile'} = \&_dl_findfile;

=back

=item *

Our C<_bootstrap> just checks if the library is in PARs

=over 4

=item *

If yes, extract it to a File::Temp temp file

=over 4

=item *

The file will be automatically cleaned up when the program ends

=back

=item *

It then pass the arguments to the original C<$bootstrap>

=item *

Finally, our C<_dl_findfile> intercepts known filenames and return it

=back

=back


=head2 Anatomy of a Self-Contained PAR executable

=over 4

=item *

The par script ($0) itself

=over 4

=item *

May be in plain-text (par.pl)

=item *

Or native executable format (par or par.exe)

=back

=item *

Any number of embedded files

=over 4

=item *

Typically used for bootstrapping PAR's various XS dependencies

=item *

Each section begins with the magic string "FILE"

=item *

Length of filename in pack('N') format and the filename (auto/.../)

=item *

File length in pack('N') and the file's content(not compressed)

=back

=item *

One PAR file

=over 4

=item *

This is just a zip file as usual

=item *

Beginning with the magic string C<"PK\003\004">

=back

=item *

Ending section

=over 4

=item *

A pack('N') number of the total length of FILE and PAR sections

=item *

Finally, there must be a 8-bytes magic string: C<"\012PAR.pm\012">

=back

=back


=head2 Self-Bootstrapping Tricks

=over 4

=item *

All we can expect is a working perl interpreter

=over 4

=item *

The self-contained script *must not* use any modules at all

=item *

Not even strict.pm or DynaLoader.pm

=item *

But to process PAR files, we need XS modules like Compress::Zlib

=item *

A chicken-egg problem

=back

=item *

Solution: bundle all module and object files needed by PAR.pm

=over 4

=item *

That's what the C<FILE> section in the previous slide is for

=item *

Load modules to memory, and write object files to disk

=item *

Then use a local @INC hook to load them on demand

=back

=item *

We want to minimize the amount of temporary files

=over 4

=item *

First, try getting PerlIO::scalar loaded

=over 4

=item *

So everything else can be in-memory

=back

=item *

Next, try getting File::Temp loaded for better C<tempfile()>

=item *

Set up an END hook to unlink all temp files up to this point

=item *

Load all other bundled files

=item *

Finally we are able to look in the compressed PAR section

=back

=item *

This can be so much easier if we have a pure-perl C<inflate()>

=over 4

=item *

Patches welcome!

=back

=back


=head1 SEE ALSO

L<http://www.autrijus.org/par-tutorial/>

L<http://www.autrijus.org/par-intro/> (English version)

L<http://www.autrijus.org/par-intro.zh/> (Chinese version)

L<PAR>, L<pp>, L<par.pl>, L<parl>

L<ex::lib::zip>, L<Acme::use::strict::with::pride>

L<App::Packer>, L<Apache::PAR>, L<CPANPLUS>, L<Module::Install>

=head1 AUTHORS

Autrijus Tang E<lt>autrijus@autrijus.orgE<gt>

L<http://par.perl.org/> is the official PAR website.  You can write
to the mailing list at E<lt>par@perl.orgE<gt>, or send an empty mail to
E<lt>par-subscribe@perl.orgE<gt> to participate in the discussion.

Please submit bug reports to E<lt>bug-par@rt.cpan.orgE<gt>.

=head1 COPYRIGHT

Copyright 2002, 2003 by Autrijus Tang E<lt>autrijus@autrijus.orgE<gt>.

This document is free documentation; you can redistribute it and/or
modify it under the same terms as Perl itself.

See L<http://www.perl.com/perl/misc/Artistic.html>

=cut
