
=head1 NAME

IO::Compress::Lzf::FAQ -- Frequently Asked Questions about IO::Compress::Lzf

=head1 DESCRIPTION

Common questions answered.

=head2 Compatibility with Unix compress/uncompress.

This module is not compatible with Unix C<compress>.

If you have the C<uncompress> program available, you can use this to read
compressed files

    open F, "uncompress -c $filename |";
    while (<F>)
    {
        ...

Alternatively, if you have the C<gunzip> program available, you can use
this to read compressed files

    open F, "gunzip -c $filename |";
    while (<F>)
    {
        ...

and this to write compress files, if you have the C<compress> program
available

    open F, "| compress -c $filename ";
    print F "data";
    ...
    close F ;

=head2 Accessing .tar.Z files

See previous FAQ item.

If the C<Archive::Tar> module is installed and either the C<uncompress> or
C<gunzip> programs are available, you can use one of these workarounds to
read C<.tar.Z> files.

Firstly with C<uncompress>

    use strict;
    use warnings;
    use Archive::Tar;

    open F, "uncompress -c $filename |";
    my $tar = Archive::Tar->new(*F);
    ...

and this with C<gunzip>

    use strict;
    use warnings;
    use Archive::Tar;

    open F, "gunzip -c $filename |";
    my $tar = Archive::Tar->new(*F);
    ...

Similarly, if the C<compress> program is available, you can use this to
write a C<.tar.Z> file

    use strict;
    use warnings;
    use Archive::Tar;
    use IO::File;

    my $fh = new IO::File "| compress -c >$filename";
    my $tar = Archive::Tar->new();
    ...
    $tar->write($fh);
    $fh->close ;

=head2 Using C<InputLength> to uncompress data embedded in a larger file/buffer.

A fairly common use-case is where compressed data is embedded in a larger
file/buffer and you want to read both.

As an example consider the structure of a zip file. This is a well-defined
file format that mixes both compressed and uncompressed sections of data in
a single file. 

For the purposes of this discussion you can think of a zip file as sequence
of compressed data streams, each of which is prefixed by an uncompressed
local header. The local header contains information about the compressed
data stream, including the name of the compressed file and, in particular,
the length of the compressed data stream. 

To illustrate how to use C<InputLength> here is a script that walks a zip
file and prints out how many lines are in each compressed file (if you
intend write code to walking through a zip file for real see
L<IO::Uncompress::Unzip/"Walking through a zip file"> ). Also, although
this example uses the zlib-based comnpresion, the technique can be used by
the other C<IO::Uncompress::*> modules.

    use strict;
    use warnings;

    use IO::File;
    use IO::Uncompress::RawInflate qw(:all);

    use constant ZIP_LOCAL_HDR_SIG  => 0x04034b50;
    use constant ZIP_LOCAL_HDR_LENGTH => 30;

    my $file = $ARGV[0] ;

    my $fh = new IO::File "<$file"
                or die "Cannot open '$file': $!\n";

    while (1)
    {
        my $sig;
        my $buffer;

        my $x ;
        ($x = $fh->read($buffer, ZIP_LOCAL_HDR_LENGTH)) == ZIP_LOCAL_HDR_LENGTH 
            or die "Truncated file: $!\n";

        my $signature = unpack ("V", substr($buffer, 0, 4));

        last unless $signature == ZIP_LOCAL_HDR_SIG;

        # Read Local Header
        my $gpFlag             = unpack ("v", substr($buffer, 6, 2));
        my $compressedMethod   = unpack ("v", substr($buffer, 8, 2));
        my $compressedLength   = unpack ("V", substr($buffer, 18, 4));
        my $uncompressedLength = unpack ("V", substr($buffer, 22, 4));
        my $filename_length    = unpack ("v", substr($buffer, 26, 2)); 
        my $extra_length       = unpack ("v", substr($buffer, 28, 2));

        my $filename ;
        $fh->read($filename, $filename_length) == $filename_length 
            or die "Truncated file\n";

        $fh->read($buffer, $extra_length) == $extra_length
            or die "Truncated file\n";

        if ($compressedMethod != 8 && $compressedMethod != 0)
        {
            warn "Skipping file '$filename' - not deflated $compressedMethod\n";
            $fh->read($buffer, $compressedLength) == $compressedLength
                or die "Truncated file\n";
            next;
        }

        if ($compressedMethod == 0 && $gpFlag & 8 == 8)
        {
            die "Streamed Stored not supported for '$filename'\n";
        }

        next if $compressedLength == 0;

        # Done reading the Local Header

        my $inf = new IO::Uncompress::RawInflate $fh,
                            Transparent => 1,
                            InputLength => $compressedLength
          or die "Cannot uncompress $file [$filename]: $RawInflateError\n"  ;

        my $line_count = 0;

        while (<$inf>)
        {
            ++ $line_count;
        }

        print "$filename: $line_count\n";
    }

The majority of the code above is concerned with reading the zip local
header data. The code that I want to focus on is at the bottom. 

    while (1) {
    
        # read local zip header data
        # get $filename
        # get $compressedLength

        my $inf = new IO::Uncompress::RawInflate $fh,
                            Transparent => 1,
                            InputLength => $compressedLength
          or die "Cannot uncompress $file [$filename]: $RawInflateError\n"  ;

        my $line_count = 0;

        while (<$inf>)
        {
            ++ $line_count;
        }

        print "$filename: $line_count\n";
    }

The call to C<IO::Uncompress::RawInflate> creates a new filehandle C<$inf>
that can be used to read from the parent filehandle C<$fh>, uncompressing
it as it goes. The use of the C<InputLength> option will guarantee that
I<at most> C<$compressedLength> bytes of compressed data will be read from
the C<$fh> filehandle (The only exception is for an error case like a
truncated file or a corrupt data stream).

This means that once RawInflate is finished C<$fh> will be left at the
byte directly after the compressed data stream. 

Now consider what the code looks like without C<InputLength> 

    while (1) {
    
        # read local zip header data
        # get $filename
        # get $compressedLength

        # read all the compressed data into $data
        read($fh, $data, $compressedLength);

        my $inf = new IO::Uncompress::RawInflate \$data,
                            Transparent => 1,
          or die "Cannot uncompress $file [$filename]: $RawInflateError\n"  ;

        my $line_count = 0;

        while (<$inf>)
        {
            ++ $line_count;
        }

        print "$filename: $line_count\n";
    }

The difference here is the addition of the temporary variable C<$data>.
This is used to store a copy of the compressed data while it is being
uncompressed.

If you know that C<$compressedLength> isn't that big then using temporary
storage won't be a problem. But if C<$compressedLength> is very large or
you are writing an application that other people will use, and so have no
idea how big C<$compressedLength> will be, it could be an issue.

Using C<InputLength> avoids the use of temporary storage and means the
application can cope with large compressed data streams.

One final point -- obviously C<InputLength> can only be used whenever you
know the length of the compressed data beforehand, like here with a zip
file. 

=head1 SEE ALSO

L<Compress::Zlib>, L<IO::Compress::Gzip>, L<IO::Uncompress::Gunzip>, L<IO::Compress::Deflate>, L<IO::Uncompress::Inflate>, L<IO::Compress::RawDeflate>, L<IO::Uncompress::RawInflate>, L<IO::Compress::Bzip2>, L<IO::Uncompress::Bunzip2>, L<IO::Compress::Lzma>, L<IO::Uncompress::UnLzma>, L<IO::Compress::Xz>, L<IO::Uncompress::UnXz>, L<IO::Compress::Lzop>, L<IO::Uncompress::UnLzop>, L<IO::Compress::Lzf>, L<IO::Uncompress::UnLzf>, L<IO::Uncompress::AnyInflate>, L<IO::Uncompress::AnyUncompress>

L<Compress::Zlib::FAQ|Compress::Zlib::FAQ>

L<File::GlobMapper|File::GlobMapper>, L<Archive::Zip|Archive::Zip>,
L<Archive::Tar|Archive::Tar>,
L<IO::Zlib|IO::Zlib>

=head1 AUTHOR

This module was written by Paul Marquess, F<pmqs@cpan.org>. 

=head1 MODIFICATION HISTORY

See the Changes file.

=head1 COPYRIGHT AND LICENSE

Copyright (c) 2005-2010 Paul Marquess. All rights reserved.

This program is free software; you can redistribute it and/or
modify it under the same terms as Perl itself.

