=pod

=head1 NAME

Set::Files - routines to work with files, each definining a single set

=head1 SYNOPSIS

  use Set::Files;
  $Version = $Set::Files::VERSION;

  $obj     = new Set::Files(OPT => VAL, OPT => VAL, ...);

  @set     = $obj->list_sets( [TYPE] );

  @uid     = $obj->owner;
  $uid     = $obj->owner(SET);

  @set     = $obj->owned_by(UID [,TYPE]);

  @ele     = $obj->members(SET);

  $flag    = $obj->is_member(SET, ELE);

  @type    = $obj->list_types( [SET] );

  @dir     = $obj->dir;
  $dir     = $obj->dir(SET);

  %opts    = $obj->opts(SET);
  $val     = $obj->opts(SET,VAR);

  $obj->cache;

  $num     = $obj->add   (SET, FORCE, COMMIT, ELE1,ELE2,...);
  $num     = $obj->remove(SET, ELE1,ELE2,...);

  $obj->commit(SET1,SET2,...);

  $obj->delete(SET);
  $obj->delete(SET,1);

=head1 DESCRIPTION

This is a module for working with simple sets of elements where each set is
defined in a separate file (one file for each set to be defined).

The advantages of putting each set in a separate file are:

=over 4

=item B<Set managment can be delegated>

If all sets are defined in a single file, management of all sets must be
done by a single user, or by using a suid program.  By putting each set in
a separate file, different files can be owned by different users so
management of different sets can be delegated.

=item B<Set files are a simple format>

Because a file consists of a single set only, there is no need to have a
complex file format which has to be parsed to get information about the
set.  As a result, set files can easily be autogenerated or edited with
any simple text editor, and errors are less likely to be introduced into
the file.

=back

The disadvantages are:

=over 4

=item B<Permissions problems>

Some applications may need to read all of the data, but since the
different set files may be owned by different people, permissions may get
set such that not all set files are readable.

Applications which actually gather all of the data will need to be run as
root in order to be reliable.  Alternately, some means of enforcing the
appropriate permissions needs to be in place.

=item B<No central data location>

Usually, when you want to define sets, the data ultimately needs to be
stored in one central location (which might be a single file or database).

To get around this, a wrapper must be written using this module to copy
the data to the central location.

=item B<Simple elements only>

Many types of sets have elements which have attributes (for example, a
ranking within the set or some other attribute).  When you start adding
attributes, you need a more complex file structure in order to store this
information, so that type of set is not addressed with this module.  The
only attribute that an element has is membership in the set.

=item B<Slow data access>

Because the data is spread out over several files, each of which must be
parsed, and any error checking done, accessing the data can be
significantly slower than if the data were stored in a central location.

=back

Features of this module include:

=over 4

=item B<Data caching>

This module provides routines for caching the information from all the set
files.  This can be used to avoid the permissions problems (allowing user
run applications access to all cached data) and decrease access time (no
parsing is left, and error checking can be done prior to caching the
information).

This still requires that a privileged user or suid script be used to
update the cache.

=item B<Multiple type of sets>

Often, it is conveniant to define different types of sets using a single
set of files as there may be considerable overlap between the sets of
different types.

For example, it might be useful to create files containing sets of users
who belong to different committees in a department.  Also, there might be
sets of users who belong to various departmental mailing lists.  One
solution is to have two different directories, one with set files with
lists of users on the various committees; one with set files with lists of
users on each mailing list.  Since there might be overlap between these
groups, it might be nice to have the two sets of files overlap.  For
example, some committees may want to have a mailing list associated with
the group, others don't want a mailing list, and there may be mailing
lists not associated with a committee.

This allows you to have a single file for each set of users, but some sets
will have mailing lists, some will be committees, and some will be both.

=item B<Set ownership>

Since the different files may be owned by different people, operations
based on set ownership can be done.

=back

=head1 METHODS

The following methods are available:

=over 4

=item B<VERSION>

  use Set::Files;
  $Version=$Set::Files::VERSION;

Check the module version.

=item B<new>

  $obj = new Set::Files(OPT => VAL, OPT => VAL, ...);

This creates a new Set::Files object which reads the appropriate set files
(or a cache of the information in set files).  The initialization options
available are described below.

=item B<list_sets>

  @set     = $obj->list_sets( [TYPE] );

Returns a list of all defined sets or the sets of the specified type.

=item B<owner>

  @uid     = $obj->owner;
  $uid     = $obj->owner(SET);

Lists all UIDs who own a set, or the owner of the specified set.

=item B<owned_by>

  @set     = $obj->owned_by(UID [,TYPE]);

Lists all sets owned by the specified UID (or those of a specific type).

=item B<members>

  @ele     = $obj->members(SET);

Lists all elements in the specified set.

=item B<is_member>

  $flag    = $obj->is_member(SET, ELE);

Returns 1 if ELE is a member of SET.

=item B<list_types>

  @type    = $obj->list_types( [SET] );

A list of all types defined, or the types that the specified set
belong to.

=item B<dir>

  @dir     = $obj->dir;
  $dir     = $obj->dir(SET);

All directories containing set files, or the directory containing the
file of the specified set.

=item B<opts>

  %opts    = $obj->opts(SET);
  $val     = $obj->opts(SET,VAR);

Returns a hash of all options set for a set, or the value of a specific
option.  If the specific option is not set, 0 is returned.

=item B<delete>

  $obj->delete($set);
  $obj->delete($set,1);

This removes the specified set file.  By default, it renames the set
file to .set_files.$set (which are ignored when reading in set data).
If the optional second argument is passed in, no backup is made (i.e.
the set file is deleted completely).

This method is only available to those who have write access to the
directory containing the set file.

=item B<cache>

  $obj->cache;

This dumps the current set information to a cache file.  This method is
only valid if the data was read in from files.  If it was read in from
the cache, this method will fail.

=item B<add, remove>

  $num = $obj->add   (SET, FORCE, COMMIT, ELE1,ELE2,...);
  $num = $obj->remove(SET, FORCE, COMMIT, ELE1,ELE2,...);

These functions add/remove the specified elements to/from the set.

When adding elements to a set, it is first checked to see if the
element is already in the set, and if so, whether it is explicitely
excluded in the set file, or comes from some other set file via.
an B<INCLUDE> tag.

If the element is not in the set, it is added.  If the B<FORCE>
flag is true, the element will be added to the set file explicitly
if it is already in the set, but only via. an B<INCLUDE> tag.  In
either case, any B<OMIT> tag which removes this element will be
removed from the list.

When removing elements from a set, a similar set of tests are done.
If the element is in the set, it is removed from the file (if it
appears in the file) AND a B<OMIT> tag is included.  If the element
does NOT appear in the set, the file is unmodified unless the B<FORCE>
flag is true, in which case an B<OMIT> tag is added.

The B<COMMIT> flag is used to determine whether the file should be
written out over the existing file.  The file can only be written out
if data was read from the files.  If it was read in from the cache,
this will fail.

The return value is the number of changes made to the set.

=item B<commit>

  $obj->commit(SET1,SET2,...);

Any changes that have been made with the B<add> and B<remove> methods
can be written out to the set file(s) with this method.  This method is
only valid if the data was read in from files. If it was read in from
the cache, this method will fail.

=back

=head1 INIT OPTIONS

The following options can be passed in to the B<new> method:

=over 4

=item B<path>

  path => DIR1:DIR2:...
  path => [ DIR1, DIR2, ... ]

The set files may be stored in one or more different directories.  By
default, set files are assumed to be in the current directory, but using
this option, the directory (or directories) can be explicitely set.

One thing to note.  If multiple directories are used, and a file of the
same name exists in more than one of the directories, the first one found
(in the order that the directories are included in the list) is used.  A
warning will be issued for files of the same name in other directories,
but they will be ignored.

Warnings will be issued for unreadable directories, or unreadable files
within a directory.

=item B<valid_file>

  valid_file => REGEXP
  valid_file => !REGEXP
  valid_file => \&FUNCTION

By default, all files in the directories are used.  With this option,
filenames are tested and only those that pass will be used.  Others will be
silently ignored.

REGEXP is a regular expression.  Only filenames which match the REGEXP will
pass (or if !REGEXP is used, only filenames which do NOT match REGEXP will
pass).

If a reference to a function is passed in, the function &FUNCTION(dir,file)
will be evaluated for each file.  If it returns 0, the file will be
silently ignored.  Otherwise it will be used.

=item B<invalid_quiet>

  invalid_quiet = 1

By default, when a file is ignored due to failing a B<valid_file> test, or
when an element is ignored due to failing a B<valid_ele> test, a warning
is issued.  With this option, no warning is issued.

=item B<cache>

  cache => DIR

Data from the set files may be cached in order to speed up data access.
If this option is used, you must specify the directory where the data
will be cached.  The directory may be the same as one of the directories
containing the set files.

The cache directory defaults to the first directory given in the B<path>
option (or the current directory if no B<path> option is given).

=item B<read>

  read => "cache"
  read => "files"
  read => "file"

When an application wants to use data from the set files, they can either
read the data from set files or the cache.

If the B<cache> option was used, the default is to read from the cache
if it exists, read from the files otherwise.  If no B<cache> option
was used, the default is to read from the files.  When data is read in
from the cache, the B<commit> and B<cache> methods are disabled.

If the B<file> option is used, it reads a single set from a single
file along with all dependancy sets (i.e. sets that are included or
excluded via. the appropriate tags).  This allows someone to make changes
to a single set file that they own even if permissions are set so that
they cannot read other set files.  The B<commit> method is available, but
the B<cache> method is disabled.  The B<file> option requires that the
B<set> option also be present.

With the B<files> option, all set files are read.  Both the B<commit>
and B<cache> methods are enabled.

=item B<set>

  set => SET

This defines which set to read when the B<read => file> option is used.
This option is required when B<read => file> and ignored for any other
value for B<read>.

=item B<types>

  types => TYPE
  types => [ TYPE1, TYPE2, ... ]

Sets can be of one or more types (or they can belong to no type and be
used solely in building other sets using the B<INCLUDE> or B<EXCLUDE>
tags described in the B<FILE FORMAT> section below).

This option can be used to specify the names of the different types
of sets defined by these files.

If this option is not given, then there is only one type and by default,
all sets belong to it.

=item B<default_types>

  default_types => [ TYPEa, TYPEb, ... ]
  default_types => "all"
  default_types => "none"
  default_typew => TYPE

Some types of sets may be more common than others, and you may or may not
want to have to explicitely define which types a set belong to.

If a list of types are passed in, every type must be defined in the
B<types> option (warnings will be issued if they weren't).  If a value
of "all" is passed in, sets belong to all types by default.  If a value
of "none" is passed in, sets don't belong to any type by default.

By default, sets belong to all types available.

=item B<comment>

  comment => REGEXP

This defines a regular expression used to recognize (and strip out) comments
from a set file.  The default expression is "#.*" which means that all
characters from a pound sign to the end of the line are removed.

If REGEXP is passed in as an empty string, there are no comments.  All
lines are either empty or contain an element.

=item B<tagchars>

  tagchars => STRING

This defines a character (or a string) which marks a line of the set
file as containing a tag.  The default value is "@".

=item B<valid_ele>

  valid_ele => REGEXP
  valid_ele => !REGEXP
  valid_ele => \&FUNCTION

By default, every non-blank line (after comments have been stripped out)
is treated as an element.  If this option is used, elements are tested,
and only those that pass the test are treated as valid.  Others are
invalid and produce a warning.

If a reference to a function is passed in, the function &FUNCTION(set,ele)
will be evaluated for each element.  If it returns 0, the element will be
silently ignored.  Otherwise it will be included in the set.

=item B<scratch>

  scratch => DIR

When automatically updating a set file, the directory where the files
live may or may not be writable by a user who owns a set file.

If the directory is writable by the user, there is no problem.  In this
case, when a new set file is written, the old one is backed up and the
new one written in it's place.

If the directory is NOT writable by the user, the old copy is backed
up to the B<scratch> directory.  This directory must be writable by the
user.  It defaults to B</tmp>.

=back

=head1 FILE FORMAT

A set file has a very simple format.  It consists of blank lines, tags,
and elements.  Comments may be included as whole lines or part of one
of the above lines.

Each line is checked for comments and they are removed before any other
processing is done.  A comment is anything that matches a regular
expression which can be set using the B<comment> Init option.  The default
regular expression is "#.*" which means that comments start with a pound
sign anywhere on the line and go to the end of the line.

Tags are lines which begin with begin with a special string (which can be
set with the B<tagchars> Init option.  The default string is "@".  Tag lines
are of one of the formats:

  @TAG
  @TAG VAL1,VAL2,...

All other lines are elements.  Elements are any string (one per line).

Leading/trailing spaces are ignored in all cases.

The set name is the name of the set file.

The following TAGs are known:

=over 4

=item B<INCLUDE SET1,SET2,...>

This includes all members of one or more other sets in the current set.

=item B<EXCLUDE SET1,SET2,...>

This excludes all members of one or more other sets from the current set.
This overrides any members included from other sets, but does NOT exclude
members explicitely included in the set file.

=item B<OMIT ELE>

This exludes a specific element from the current set.  This overrides
any elements included via. an B<INCLUDE> tag, or any elements explicitly
included in the set file.

Each element must be specified separately since there is no guarantee that
elements may not contain commas.

=item B<TYPE TYPE1,TYPE2,...>

The default types that this set belongs to are determined by the B<types>
and B<default_types> Init options.

This tag explicitely puts this set if the specified types, even if it
is not in those types of default.

=item B<NOTYPE TYPE1,TYPE2,....>

Similar to the B<TYPE> tag, but this tag explicitely removes the set
from the specified types, even if it is in them by default.

=item B<OPTION VARIABLE [= VALUE]>

Although there is no support for element specific attributes, there IS
support for attributes which apply to the entire set (and which can be made
available to applications using these sets).

Each set may have a hash associated with with key/value pairs (if no value
is include, it defaults to 1).  These attributes are available using the
B<info> method.

=back

All tag lines can be repeated any number of times, so:

  @INCLUDE foo,bar

is equivalent to

  @INCLUDE foo
  @INCLUDE bar

All tags are case insensitive.

When determining the members of a set which includes and excludes other
sets, or omits specific elements from the set, all inclusions are evaluted
first, followed by all exclusions (i.e. all exclusions override all
inclusions).  If there is a cyclic dependancy (i.e. A depends on B depends
on A where a dependancy can either be an B<INCLUDE> or B<EXCLUDE>), an
error is reported and the cyclic dependancy is ignored.

A few examples illustrate the use of B<INCLUDE>, B<EXCLUDE>, and B<OMIT>
tags.  In the examples, the set file B<A> contains the elements: B<E1, E2,
E3>.  The set file B<B> contains the elements: B<E3, E4, E5>.  The set file
contains the following lines:

  @INCLUDE A
  @EXCLUDE B
  E5
  E6

defines a set contains the elements: B<E1, E2, E5, E6>.  The first line
includes B<E1, E2, E3>.  The second line excludes B<E3>.  It does NOT
exclude B<E5> since the B<EXCLUDE> tag does not override elements
explicitly included in the set file.  Finally, the B<E5> and B<E6> elements
are added.

The set file containing the following lines:

  @INCLUDE A
  @EXCLUDE B
  @OMIT    E2
  @OMIT    E6
  E5
  E6

defines a set contains the elements: B<E1, E5>.  This is similar to the
above example, except that the B<OMIT> tags override elements included
via. the B<INCLUDE> tag AND elements explicitly included in the set file.

=head1 FILES

Several files are used by the B<Set::Files> module.  They all live in the
directory set by the B<cache> Init Option except for set specific files
which live in the same directory as the set file.  Files are:

=over 4

=item B<.set_files.SET>

A backup of the given set.  When a set file is updated, the original file
is stored in this file.  The file is stored either in the same directory
as the set file (if it is writable) or in the directory specified by the
B<scratch> Init Option.

=item B<.set_files.SET.new>

A temporary file where a new set file (or the update to an old one) is
written.  Once completed, this file is moved into place as the new set
file.  This file lives in the same directory as the set file or in the
B<scratch> directory.

=item B<.set_files.cache>

The file containing the cache.  This is created using the B<cache> method.

=item B<.set_files.template>

When creating a new set file (or updating an existing one), this file is
used (if it exists) as a starting point and then all the data is appended
to it.  This is a good place to store comments describin how to edit
the set files, etc., that set file maintainers can read for help.

=back

=head1 KNOWN PROBLEMS

None at this point.

=head1 LICENSE

This script is free software; you can redistribute it and/or
modify it under the same terms as Perl itself.

=head1 AUTHOR

Sullivan Beck (sbeck@cpan.org)

=cut
