package HPCI;
### HPCI.pm ###################################################################

### HISTORY ###################################################################
# Version       Date        Developer         Comments
# 0.30          2015-04-09  John Macdonald    preparing initial CPAN releaase
# 0.31          2015-05-29  John Macdonald    initial CPAN release

### INCLUDES ##################################################################

# safe Perl
use warnings;
use strict;
use Carp;
use Module::Load;

sub group {
    my $pkg = shift;
    my $args =
        scalar(@_) == 1 && ref($_[0]) eq 'HASH' ? shift
        : scalar(@_) % 2 == 0                   ? { @_ }
        : croak("HPCI->group() requires a hashref or a hash in list form");
    my $cluster = $args->{cluster} || croak("HPCI->group() requires a cluster key in the argument hash");
    $args->{log_file} //= 'group.log';
    my $clmod = "HPCD::${cluster}::Group";
    load $clmod;
    return $clmod->new($args);
}


sub _trigger_mkdir {
    my $self = shift; # an object with a log
    my $dir  = shift; # a Path::Class::Dir object
    $self->info( "Created directory: $_" ) for $dir->mkpath;
}

=head1 NAME

    HPCI

=head1 VERSION

Version 0.31

=cut

our $VERSION = '0.31';

=head1 SYNOPSIS

    use HPCI;

    my $group = HPCI->group(
        cluster => ($ENV{HPCI_CLUSTER} // 'uni'),
        ...
    );
    $group->stage(
        name => 'analysis_A',
        command => '...'
    );
    $group->stage(
        name => 'analysis_B',
        command => '...'
    );
    $group->stage(
        name => 'analysis_C',
        command => '...'
    );
    $group->stage(
        name => 'report',
        command => '...'
    );
    $group->add_deps(
        pre_reqs => [ qw(analysis_A analysis_B analysis_C) ],
        dep => 'report'
    );
    my $status_info = $group->execute;

    my $exit_status = 0;
    for my $stage ( qw(analysis_A analysis_B analysis_C report) ) {
        if (my $stat = $status_info->{$stage}[-1]{exit_status}) {
            $exit_status ||= $stat;
            print stderr "Stage $stage failed, status $stat!\n";
        }
    }

=head1 OVERVIEW

HPCI (High Performance Computing Interface) provides an interface to
a range of types of computer aggregations (clusters, clouds, ...).
(The rest of this document will use I<cluster> henceforth to refer
to any type of aggregation that is supported by HPCI.)

A cluster is defined as a software interface that allows running
multiple programs on separate compute elements (nodes).

HPCI uses an HPCD (High Performance Computing Driver) module
to translate its standard interface into the appropriate access
mechanisms for the type of cluster that is selected.  (If you have
used the DBI/DBD modules for accessing databases, this will seem
very familiar.)

The goal of this HPCI/HPCD split is to allow users to write
programs that make use of cluster facilities in a portable manner.
If there is a reason to run the same program using a different
type of cluster, it should only require change the cluster
definition attributes provided to one parent object creation; the
rest of code need not know or care about the changed cluster type.
Programs which are likely to be run on different cluster types will
usually be written to get the cluster attribute information from
a configuration file, or command line arguments - so the program
itself need not change at all.

Running a program on different types of clusters can happen for a
number of reasons.  An organization might have access to multiple
types of cluster, such as an in-house cluster plus an external cloud.
Scholarly research often shares programs both to allow similar
research, or to validate existing research results.

HPCD modules can provide cluster-specific extensions.  That can
either be a different kind of functionality, or it can be as simple
as allowing the teminology familiar to users of that cluster type
to be used in place of the generic terminology provided by HPCI.
However, using such extensions makes it harder to move to a
different cluster type.  So, actually making use of such extensions
must be considered carefully.

=head1 INTERFACE

The life cycle of running a group of commands on a cluster is as follows:

=over

=item create group

A B<group> object is created using the HPCI "class method" B<group>.
HPCI isn't really a class, it just appears to be one.  Its B<group>
"class method" actually delegates creation of a group object to
the HPCD module that is indicated by the I<cluster> attribute
and it returns an cluster-specific group object that supports the
HPCI interface.

=item create stages

A B<stage> is created for each command that is to be executed on a
separate node of the cluster.  This is created using the B<group>
object's method B<stage>.

=item define dependency ordering

An important reason for running a group of jobs on a cluster is the
ability to use multiple computers to run portions of the computation
at the same time, rather than having them compete for the rsources
of a single computer.  However, often some stages will depend
upon the output of other stages.  Such a dependent stage cannot
start executing until all pre-requisite stages have completed.
Specifying such dependency requirements is done with the B<group>
method B<add_deps>.

=item execution

Finally, the B<group> method B<execute> will run the entire set
of stages.  It does not return until all stages have completed (or
have been skipped).  Each stage will normally be run once.  However,
it is possible for there to be retries under some recoverable
failure conditions causing some stages to be run .multiple times.
Alternatively, a hard failure of one stage can be a trigger for
completely skipping the execution of other stages.  Each separate
execution of a stage is managed with an internal object (called a
job - but a user program won't see job objects directly).

=back

=head2 OUTPUT DIRECTORY LAYOUT

There are a number of output files created during a group execution.

The default layout of these files is:

  <base_dir>                     "."
    <group_dir>                  <base_dir>/<name>-<YYYYMMDD-hhmmss>
      <log>                      <group_dir>/<name>.log
      <stage_dir>                <group_dir>/<stage_name>
        <script_file>            <stage_dir>/script.sh
        <job_dir>                <stage_dir>/<retry_number>
          stdout
          stderr
        final_retry              symlink to final <job_dir>

Many of these files/directories can be re-assigned to different
location using group or stage attributes - shown avove is the
default layout.  Commonly, you will specifically use the I<base_dir>
attribute to choose a location other than the current directory for
placing the tree; or else use the I<group_dir> attribute if you want
to choose a location that does not create a sub-directory for you.
(If this is an already existing directory that is being re-used you
may end up with a mixture of old and new contents that are hard to
figure out.)

=head3 base_dir

The top level of all the generated output.  It defaults to ".",
but can be specified explicitly when the group is created.

=head3 group_dir

By default, a new directory is created under B<base_dir>.  Its name
is I<name>-I<YYYYMMDD>-I<hhmmss> - the name of the group along with
a timestamp of when the execution started.

=head3 log

The automatically provided log is written to the file I<name>.log
directly under I<group_dir>.  This logs information about the
execution of the entire group of stages.

=head3 stage_dir

Each stage creates a sub-directory beneath I<group_dir> with the
same name as the stage.

=head3 script_file

The script created to be executed on the cluster node.  This wraps
the specified command with additional logic to pass on environment
and config info, and to set output redirection.  It is called
"script.sh" and placed in I<stage_dir>.

=head3 job_dir

A sub-directory is created under I<stage_dir> for each attempt to
run the command.  Usually, there will only be a single attempt.
However, if the cluster driver provides mechanisms for detecting
recoverable issues and then retries a command there can be more
than one attempt; or alternately, if a pre-requisite stage
fails there might be no attempt made (in that case, though,
the entire I<stage_dir> directory would not even get created).
These directories are simply named with the retry number ("0",
"1", ...).

=head3 stdout/stderr

Within each I<job_dir>, the files "stdout" and "stderr" collect
the standard output and standard error output from that (re)try
attempt to run the command.

=head3 final_retry

A symlink named "final_retry" is created within I<stage_dir> that
points to the I<job_dir> of the final (re)try.  Since you often
don't care as much about the initial run tries as you do about the
last one, this symlink provides a consistant access path to that
final retry.

=head2 HPCI "Class"

You can pretend that B<HPCI> is a class with one class method
named B<group>.

=head3 METHOD group

The B<group> method creates and returns a group object, which
you can treat like a B<HPCI::Group> object.  (In fact, it really
returns an object of class B<HPCD::I<cluster>::Group>, but if you
ignore that fact then you can trivially have your program run on
some other cluster type.)

The group object is described further below.

=head2 Group Object

A group object is created with the HPCI class method B<group>
(and not using a B<new> method directly.

=head3 Cluster Attributes

The one necessary attribute is B<cluster>.  For some specific
cluster types there may be additional attributes required for
connecting to the cluster software (authentification, usage
class info, etc.).

=head4 cluster

The cluster attribute specifies which type of cluster is to be used.
This is the only required attribute.  (Some cluster types may have
additional attributes that are required for specifying connection
and authentification info.)

=head3 Basic Attributes

=head4 name

The name you give to a group is used for creating the directory
where output is stored, and also in log messages.  A default name
"default_group_name" is provided if you do not specific an explicit
name.  Using the default name is adequate in simple programs which
only create one group, but for more complicated programs giving
separate names to each group is necessary to easily identify the
output of each group.

=head3 Directory Layout Attributes

=head4 base_dir

If none of the other directory layout attributes are used to
over-ride this, this attribute specifies the directory in which
all output directories and files will be created.  This is
usually an existing directory; it defaults to the current
directort ".".

=head4 group_dir

This directory is usually created to contain the outputs of the
group execution.  By default, it is directly under B<base_dir> with
a name that consists of the group name attribute and a timestamp
(e.g. "T_Definition-20150521-153256").

If you provide an explicit value for this parameter, then it
should not be an existing directory containing previous results.
(If it is, the log file will be appended to the previous one, but
the stage directories will over-write equivalently named directories
and files that are created in this run, while leaving unchanged any
that did not recur, so you'll have a mix of old and new contents.)

=head3 Logging Attributes

An HPCI group logs its activities using a Log::Log4perl logger.
The logger can either be provided by the caller, or else HPCI will
create its own.

=head4 log

This a Log::Log4perl::Logger object.  If it is provided as an
attribute to the B<group> creation call, it will be used as it is,
and the other logging attributes will be ignored.

If it is not provided by the user, a new Log::Log4perl::Logger
object will be created using the attributes below to define where
it is logged to.  This created logger will send all log entries to
a file, as well as sending all info and higher log entries to stderr.

=head4 log_path

If this attribute is provided (and the B<log> attribute is not
provided) it will be used as the full pathname of a file where the
log will be written.  If it is not provided, it will use the path
B<log_dir>/B<log_file> by default.

=head4 log_dir

If neither B<log> or B<log_path> is provided, this attribute can
be used to specify the directory where the log file is to be written.
By default, it uses B<group_dir>.

=head4 log_file

If neither B<log> or B<log_path> is provided, this attribute can
be used to specify the file name to be written in the log directory.
By default, it uses I<name>.log.

=head4 log_level

You can provide this attribute to change the default log level setting from "info" to any of I<debug info warn error fatal>.

=head3 Operational Attributes

=head4 max_concurrent

This attribute specifies the maximum number of stages that will
be executing at one time.  The default setting of 0 allows as
many stages as possible (all those that are not waiting for a
pre-requisite stage to complete) to run at the same time.

=head4 status

This attribute is set internally while stages are executed.
It contains the final result status from each stage run that
has completed.  The B<execute> method returns this value when
execution completes, so you will usually not need to access it
explicitly yourself.

This value is a hashref (indexed by stage name).  The values are
arrayrefs (indexed by run number 0..n).  For each run, there is
a hash.  The key B<exit_status> contains the exit status of the run.
If the stage was never run, B<exit_status> instead contains a text
message listing the reason that it was skipped.

=head3 Environment Passing Attributes

You can set up a set of enviroment variables that will be provided to
all stages.  (You can also set variables that are only for individual
stages - if so, they will modfy any set you provide in the group.)

See B<HPCI::Env> for a description of these.

=head3 Methods

=head4 stage

The method B<stage> is used to create a new stage object.
Its attributes are described below.

=head4 add_deps

The method B<add_deps> is used to specify pre-requisite/dependent
relationships.  It takes either a hashref or a list containing
two pairs.  One key must be either B<pre_req> or B<pre_reqs>,
the other must be either B<dep> or B<deps>.

If the singular key is used (either B<pre_req> or B<dep>), then
the corresponding value must be a scalar - either the string name
of a stage object, or a reference to a stage object.

If the plural key is used (either B<pre_reqs> or B<deps>), then the
corresponding value must be an arrayref containing a list of scalars,
each one being either the string name of a stage or a reference to
a stage object.

The plural forms are provided for convenience - often the output
file from one preparation stage is required by many others, or the
output from many processing stages is needed by a stage that merges
results into a summary report.

It is an error to provide a sequence of dependencies that form
a cycle.

=head4 execute

The B<execute> method is the final goal of building the group.
It schedules the execution of individual stages.  It allows for
dependencies before running a stage.  It provides for re-running
a stage if a soft failure has occurred that allows a retry.  If a
failure that cannot be retried occurrs, it skips scheduling dependent
stages, or stops scheduling all new stages.

=head2 Stage Object

=head3 Attributes

=head4 name

The B<name> attribute must be provided for stages.  It is a string.
Each stage within a group must have a unique name.  There is no
default value provided.

=head4 command

The B<command> attribute must be provided before the group is
executed.  It can either be provided as a string attribute in the
B<stage> method call that creates the stage, or using the one of
the command-setting methods provided by the stage class.

See B<HPCI::Stage> for more details about the command setting
methods.

=head4 dir

The B<dir> attribute is optional.  It specifies the direcory
in which files related to the stage are placed.  By default,
it is I<group_dir>/I<stage_name>.  You will usually not need to
change this.

=head4 resources_required

=head4 retry_resources_required

The B<resources_required> and B<retry_resources_required> are used to
define resources that will be required by the stage when it executes.
These attributes are somewhat cluster specific - each cluster has
its own set of requirements for how a job submission must specify
the sort of resources that it will require.

The B<resources_required> attribute is a hash, specifying the
value for each resource that is to be considered.

The B<retry_resources_required> attribute is also a hash.  For
each resource, you can specify an array of values.  If the cluster
driver is able to detect that a run failed because the resource
was inadequate, it will retry the run with the next larger value
from this list.

See B<HPCI::Stage> for more details about resources.

=head4 failure_action ('abort_group', 'abort_deps'*, or 'ignore')

Specifies the action to take if this stage fails (terminates with
a non-zero status).

There are three string values that it can have:

=over

=item - abort_deps (default)

If the stage fails, then any stages which depend upon it
(recursively) are not run.  The group continues executing until
all stages which are not dependent upon this stage (including those
that have not yet been initiated) complete execution.

=item - abort_group

If the stage fails, then no other stages are started.  The group
simply waits until stages that have already been started complete
and then returns.

=item - ignore

Execution continues unchanged, any dependent stages will be run when they are
no longer blocked.

=back

As an alternative to providing a value to the failute_action attribute
when you create a stage, you can instead provide one of the pseudo-attributes
'abort_group_on_failure', 'abort_deps_on_failure', or 'ignore_failure' with
a true value to specify 'abort_group', 'abort_deps', or 'ignore' respectively.

=head3 Methods

=head4 command creation

See B<HPCI::Stage> for details.

=head3 Env

See B<HPCI::Env> for details.

=head1 Additional

This is an early public release of HPCI, and at present, there are
only two drivers available.

Only one cluster type is directly included within the HPCI package.
The cluster type B<HPCD::uni> runs on a "cluster" of only one
machine.  It simply uses fork to submit individual stages and has
facility for retries and timeouts.  This is the default cluster
type used for testing, as it will work natively on all types of
Unix systems.  It is also possible to use this driver as a fallback,
in cases where the only available "real" cluster is not accessable
for some reason.

Additionally, there is the B<HPCD::SGE> driver available on CPAN.
It has seen heavy use within Boutros Lab.

Now that these packages have been released, it is likely new
cluster drivers will be written.  People interested in developing
drivers for additional cluster types should contact the authors
of this package to co-ordinate releases, features needed, etc. at
B<mailto:BoutrosLabSoftware@oicr.on.ca>.

Additionally, you may wish to subscribe to the email list mentioned
at B<https:://lists.oicr.on.ca/mailman/listinfo/hpci-discuss>.
This is expected to be a low volume discussion group, although the
future will tell what the actual volume will be.

As additional capabilities of new cluster types are addressed, and as
different control needs used at other organizations are identified;
this interface will surely change.  As far as possible, such changes
will be done in an upwardly compatible manner, but until a few more
drivers have been integrated there is the possibility of changes
that are not fully backward compatible.  Watch the release notes
for warnings of such issues.  At some point there will be a 1.0.0
release, at which point this expectation of (limited) incompatible
future change will be dropped.  After that point, incompatible
changes will only be made for critical reasons.

The reason for separate distribution of cluster-specific HPCD
packages are fairly obvious:

=over

=item -

The maintainers of the HPCI package do not have access to every
possible cluster type, and it unlikely that anyone will have access
to all supported cluster types from one location, so the driver
modules will need to be tested separately anyhow.

=item -

A user of HPCI is equally not going to have need to access every
type of cluster that exists, so they will probably prefer to only
download the driver modules that they actually need.

=back

=head1 SEE ALSO

=over

=item HPCI::Group

Describes the interface common to all B<HPCI Group>
objects, regardless of the particular type of cluster that
is actually being used to run the stages.  In the future, the
common interface may change somewhat as supprt for additional
cluster types is added and a better understanding of the common
features is achieved.

=item HPCI::Stage

Describes the interface common to stage object returned
by all B<HPCI Stage> objects, regardless of the
particular type of cluster that is actually being used to
run the stages.  The common interface may change somewhat
as supprt for additional cluster types is added and a better
understanding of the common features is achieved.

=item HPCI::Logger

Describes the logger parameters in more detail.

=item HPCI::Env

Describes the environment passing parameters in more detail.

=item HPCD::I<$cluster>::Group

Describes the group interface unique to a specific type of cluster,
including any limitations or extensions to the generic interface.

=item HPCD::I<$cluster>::Stage

Describes the stage interface unique to a specific type of cluster,
including any limitations or extensions to the generic interface.

=back

=head1 AUTHOR

Christopher Lalansingh - Boutros Lab

John Macdonald         - Boutros Lab

=head1 ACKNOWLEDGEMENTS

Paul Boutros, Phd, PI - Boutros Lab

The Ontario Institute for Cancer Research

=cut

1;

