arclog Version 3.00
Author: imacat
Keywords: archive, Apache, Syslog, log files

正體中文 | 简体中文 | English

:::

# `arclog` Version 3.00

## What’s New?

2007-11-28

  `arclog` version 3.00 is released. The code is completely new borrowed from `reslog`, with the name changed from `arclog.pl` to `arclog`, too. The new object-oriented handlers deal with different compression methods, formats nicely. Perl’s s build system `ExtUtils::MakeMaker` and `Module::Build` are used instead of GNU `autoconf`. And there is a complete test suite that help quality assurance. Download `arclog` version 3.00.

2002-09-26

  `arclog.pl` version 2.1.0 is released. This is the final 2.1.0 release. It adds support for `bzip2` compression, `gzip` compression with `gzip` binary instead of `Compress::Zlib`, and GNU `autoconf` configuration script. Download `arclog.pl` version 2.1.0.

2002-09-18

  `arclog.pl` version 2.1.0dev2 is released. This release corrects several documentation errors. It also adds SourceForge in the documentations as one of the sources of `arclog.pl`. You can think of it as a Source Forge Memorial Release. :p You don’t have to upgrade to this version in a rush. Download `arclog.pl` version 2.1.0dev2.

2002-09-17

  `arclog.pl` is hosted at SourceForge now! Congratulations! (Although I’m still trying hard to get it working at this time… ^^; )

## Description

`arclog` archives the log files monthly. It strips off previous months’ log records from the log file, and save them to compressed archive files named `logfile.yyyymm`. It then saves the hard disk space and prevents potential attacks on log files.

Currently, `arclog` supports Apache access log, Syslog, NTP, Apache 1 SSL engine log and my own bracketed, modified ISO date/time log file formats, and `gzip` and `bzip2` compression methods.

## Caution

- Archiving takes time. To reduce the time occupying the source log file, `arclog` copies the content of the source log file to a temporary working file and restart the source log file first. Then `arclog` can take its time working on the temporary working file. However, please note:

  1. If you have a huge log file (several hundreds of MBs), merely copying still takes a lot of time. You had better stop logging first, archive the log file and restart logging, to avoid racing condition in writing. If you archive the log file periodly, it shall not grow too big.

  2. If`arclog` stops in the middle of the execution, it will leave a temporary working file. The next time `arclog` runs, it will stop when it sees that temporary working file. You have to process that temporary working file first. That temporary working file is merely a copy of the original log file. You can rename and archive it like an ordinary log file to solve this.

- Do not sort unless you have a particular reason. Sorting has the following potential problem:

  1. Sorting may eat huge memory on large log files. The amount of the memory required depends on the number of records in each archived month. Modern Linux and MS-Windows have memory consuming protection by killing processes that eats too much memory, but it still takes minutes, and your system will hang during that time. I do not know the memory consuming protection on other operating systems. If you try, you are at your own risk.

  2. The time units of all recognized log formats are second. Log records happen in a same second will be sorted by the log file order (if you are archiving several log files at a time) and then the log record order. I try to ensure that the sorted archived records are in a correct order of the happening events, but I cannot guarantee. You have to watch out if the order in a second is important.

- Be careful on the Syslog log files: Syslog does not record the year. `arclog` uses `Date::Parse` to parse the date, which assums the year between this month and last next month if the year is missing. For ex., if today is 2001-06-08, it will then assume the year between 2001-06-30 back to 2000-07-01. I think this is smart enough. However, if you do have a Syslog log file that has records older than one year, do not use `arclog`. It will destroy your log file.

- If read from `STDIN`, please note:

  1. You must specify the output prefix if you want to read from `STDIN`, since what it needs is an output pathname prefix, not an output file.

  2. `STDIN` cannot be deleted, restarted or partially kept. If you read from `STDIN`, the keep mode will fall back to keep all. If you archive several source log files including `STDIN`, the keep mode will fall back to keep all for all source log files, to prevent disaster.

  3. The answers of the `ask` mode is obtained from `STDIN`, too. Since you have only one `STDIN`, you cannot specify the `ask` mode while reading from `STDIN`. It will fall back to the `fail` mode in that case.

- I suggest you to install `File::MMagic` instead of counting on the `file` executable. The internal magic file of `File::MMagic` seems to work better than the `file` executable. `arclog` treats everything not `gzip` nor `bzip2` compressed as plain text. When a compressed log file is wrongly recognized as an image, `arclog` will treat it as plain text, read log records directly from it and fail. This failure does not hurt the source log files, but is still annoying.

## System Requirement

1. Perl, version 5.8.0 or above. `arclog` uses 3-argument `open()` to duplicate filehandles, which is only supported since 5.8.0. I have not successfully port this onto earlier versions yet. Please tell me if you made it. You can run `perl -v` to see your current Perl version. If you do not have Perl, or if you have an older version of Perl, you can download and install/upgrade it from Perl website. If you are using MS-Windows, you can download and install ActiveState ActivePerl.

2. Required Perl modules:

   `Date::Parse`

     This is used to parse the timestamp of the log records. ActivePerl MS-Windows users can install this using `ppm install Date::Parse`.

   You can always search, download and install the missing Perl modules from the the CPAN archive.

3. Optional Perl modules:

   `File::MMagic`

     This is used to check the file type. If this is not available, `arclog` will look for the `file` executable instead. If that is not available, too, `arclog` will judge the file type by its name suffix (extension). In that case `arclog` will fail when reading from `STDIN`. ActivePerl MS-Windows users can install this using `ppm install File::MMagic`, or get `file.exe` from the GnuWin32 home page. Be sure to save it as `file.exe` somewhere in your `PATH`.

     I suggest you use `File::MMagic`. `file` executable seems to make mistakes occationally.

   `Compress::Zlib`

     This is used to support read/write of gzip compressed files. It is only needed when gzip compressed files are encountered. If it is not available when needed, `arclog` will try to use the `gzip` executable instead. If that is not available, too, `arclog` will fail. ActivePerl MS-Windows users can install this using `ppm install Compress::Zlib`, or get `gzip.exe` from the gzip home page. Be sure to save it as `gzip.exe` somewhere in your `PATH`.

   `Compress::Bzip2`, version 2 or above

     This is used to support read/write of bzip2 compressed files. It is only needed when bzip2 compressed files are encountered. If it is not available when needed, `arclog` will try to use the `bzip2` executable instead. If that is not available, too, `arclog` will fail. Notice that older versions before 2 does not work, since file I/O compression were not implemented yet. ActivePerl MS-Windows does not have `Compress::Bzip2` in their PPM deposit yet, as the time I’m writing this. You can get `bzip.exe` from the bzip home page instead. Be sure to save it as `bzip2.exe` somewhere in your `PATH`.

   `Term::ReadKey`

     This is used to display the progress bar. Without this `arclog` won’t display the progress bar, but nothing else is different. The progress bar is a good visual representation of what `arclog` is currently doing.

   You can always search, download and install the missing Perl modules from the the CPAN archive.

## Download

`arclog`’s official websites is at…

- `arclog` web site at SourceForge

- `arclog` project page at SourceForge

- `arclog` project page at CPAN

- `arclog` page at Tavern IMACAT’s

You can always download the newest version of `arclog` from…

- `arclog` download page at SourceForge

- `arclog`’s project page at CPAN

- Tavern IMACAT’s FTP directory

imacat’s PGP public key is at…

- imacat’s PGP key at `arclog` SourceForge web site

- imacat’s PGP key at Tavern IMACAT’s

## Install

If you are upgrading from `arclog.pl` 2.1.1dev4 or earlier, please read `UPGRADE` for some upgrade instruction.

### Install with `ExtUtils::MakeMaker`

`arclog` uses standard Perl installation with `ExtUtils::MakeMaker`. Follow these steps:

>     % perl Makefile.PL
>     % make
>     % make test
>     % make install

When running `make install`, make sure you have the priviledge to write to the instalation location. This usually requires the `root` priviledge.

If you are using ActivePerl under MS-Windows, you should use `nmake` instead of `make`. `nmake` can be obtained from the Microsoft FTP site.

If you want to install into another location, you can set the `PREFIX`. For example, to install into your home when you are not `root`:

>     % perl Makefile.PL PREFIX=/home/jessica

Refer to the docuemntation of `ExtUtils::MakeMaker` for more installation options (by running `perldoc ExtUtils::MakeMaker`).

### Install with `Module::Build`

You can install with `Module::Build` instead, if you prefer. Follow these steps:

>     % perl Build.PL
>     % ./Build
>     % ./Build test
>     % ./Build install

When running `./Build install`, make sure you have the priviledge to write to the instalation location. This usually requires the `root` priviledge.

If you want to install into another location, you can set the `--prefix`. For example, to install into your home when you are not `root`:

>     % perl Build.PL --prefix=/home/jessica

Refer to the docuemntation of `Module::Build` for more installation options (by running `perldoc Module::Build`).

## Options

    ./arclog [options] logfile… [output]
    ./arclog [-h|-v]

logfile

  The log file to be archived. Specify `-` to read from `STDIN`. Multiple log files are supported. `gzip` or `bzip2` compressed files are supported, too.

output

  The prefix of the output files. The output files will be named as `output.yyyymm`, i.e., `output.200101`, `output.200102`. If not specified, the default is the same as the log file. You must specify this if you want to read from `STDIN`. You cannot specify `-` (`STDIN`), since this is only a name prefix, not the output file.

-c, --compress method

  Specify the compression method for the archived files. Log files usually have large number of simular lines. Compress them saves you lots of disk spaces. (And this is why we want to archive them.) Currently the following compression methods are supported:

  g, gzip

    Compress with `gzip`. This is the default. `arclog` can use `Compress::Zlib` to compress instead of calling `gzip`. This can be safer and faster for not calling foreign binaries. But if `Compress::Zlib` is not installed, it will try to use `gzip` binary instead. If `gzip` binary is not available, either, but `gzip` compression is required, the program will fail.

  b, bzip2

    Compress with `bzip2`. `arclog` can use `Compress::Bzip2` to compress instead of calling `bzip2`. This can be safer and faster for not calling foreign binaries. But if `Compress::Bzip2` is not installed, it will try to use `bzip2` binary instead. If `bzip2` binary is not available, either, but `bzip2` compression is required, the program will fail.

  n, none

    No compression at all. (Why? :p)

--nocompress

  Do not compress the archived files. This is equal to `--compress none`.

-s, --sort

  Sort the records by time (and then the record order). Sorting eats huge memory and CPU, so it is disabled by default. See the description above for a detailed illustration on sorting.

--nosort

  Do not sort the records. This is the default.

-o, --override mode

  Whether we should overwrite the existing archived files. Currently the following modes are supported:

  o, overwrite

    Overwrite existing target files. You will lost these existing records. Use with care. This is helpful if you are sure the master log file has the most complete records.

  a, append

    Append the records to the existing target files. You may destroy the log file completely by putting irrelevant entries altogether accidently. Use with care. This is helpful if you append want to merge 2 or more log files, for ex., 2 log files of different periods.

  i, ignore

    Ignore any existing target file, and discard all the records of those months. You will lost these log records. Use with care. This is helpful if you are supplying log records for the missing months, or if you are merging the log records in a complex manner.

  f, fail

    Stop processing whenever a target file exists, to prevent destroying existing files by accident. This should be mostly wanted when run from some automatic mechanism, like `crontab`. So, this is the default if no terminal is found at `STDIN`.

  ask

    Ask you what to do when a target file exists. This should be most wanted if you are running `arclog` interactively. So, this is the default if a terminal is found at `STDIN`. The answers are read from `STDIN`. Since you have only one `STDIN`, you cannot specify this mode if you want read the log file from `STDIN`. In that case, it will fall back to the `fail` mode. Also, if `arclog` cannot get its answer from `STDIN`, for ex., on a closed `STDIN` like `crontab`, it will fall back to `fail` mode.

-k, --keep mode

  What to keep in the source file. Current the following modes are supported:

  a, all

    Keep the source file after records are archived.

  r, restart

    Restart the source log file after records are archived.

  d, delete

    Delete the source log file after records are archived.

  t, this-month

    Archive and strip records of previous months off from the log file. Keep the records of this month in the source log file, to be archived next month. This is designed to be run from `crontab` monthly, so this is the default.

-d, --debug

  Show the detailed debugging messages.

-q, --quiet

  Shihhhhhh. Only yell when errors.

-h, --help

  Display the help message and exit.

-v, --version

  Output version information and exit.

## Copyright

Copyright © 2001-2007 imacat..

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but without any warranty; without even the implied warranty of merchantability or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see `<http://www.gnu.org/licenses/>`.

## Release Notes

Please read the `Changes` for the new functions and bug fixes.

## Support

`arclog` is hosted on SourceForge, CPAN and Tavern IMACAT’s. For the latest infomation, see

- `arclog` web site at SourceForge.

- `arclog` project page at SourceForge.

- `arclog` project page at CPAN.

- `arclog` page at Tavern IMACAT’s.

There is a `arclog` mailing list hosted at SourceForge. Please submit your questions, suggestions or bug reports there. It is a Mailman mailing list. For more information, see the `arclog` mailing list page. Alternatively, you can send a mail to the e-mail command mailbox with the subject `help` for a list of available e-mail commands.