nrun
====

nrun will run a single command or script on a multiple of target servers
synchronously.  ncopy will copy a file or directory to multiple target servers.

the underlying remote access mechanism is exchangeable. as of now, ssh, nsh, rsh
and local execution modes are implemented.

the return code and all command output will be logged.

Copyright 2013 Timo Benk <benk@b1-systems.de>

nrun is free software: you can redistribute it and/or modify it under the terms
of the GNU General Public License as published by the Free Software Foundation,
either version 3 of the License, or (at your option) any later version.

nrun is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
PURPOSE.  See the GNU General Public License for more details.

installation
============

installation is straight forward:

# perl Makefile.PL
# make install

if desired it is possible to change the installation prefix using the
parameter INSTALLATION_BASE:

# perl Makefile.PL INSTALLATION_BASE=~/applications

usage
=====

usage: nrun -t HOST1[,HOST2,...] -- COMMAND
--async,-a               asynchronous output mode.
--copy,-c                copy command to target host before execution.
--dump-results,-d        instead of dumping the command output, dump the exit status.
--log-directory,-l DIR   base directory for the log files.
--no-hostname            omit hostname prefix.
--no-logfile             do not generate any log files.
--parallel,-p MAX        number of parallel connections (defaults to 5).
--skip-ns-check          skip checking if the hostname is resolvable.
--skip-ping-check        skip checking if the host answers on ping.
--target,-t HOST1,HOST2  comma separated list of target hosts.
--timeout SEC            timeout for each command execution (defaults to 60).
--version,-v             print the version string and exit.
--mode,-m MODE           remote execution mode:
                         rsh - rsh based remote execution
                         nsh - nsh based remote execution
                         ssh - ssh based remote execution
                         local - execute the script locally, set TARGET_HOST on each execution

usage: ncopy -t HOST1[,HOST2,...] -- SRC DST
--async,-a               asynchronous output mode.
--dump-output,-d         instead of dumping the exit status, dump the command output.
--log-directory,-l DIR   base directory for the log files.
--no-hostname            omit hostname prefix.
--no-logfile             do not generate any log files.
--parallel,-p MAX        number of parallel connections (defaults to 5).
--skip-ping-check        skip checking if the host answers on ping.
--skip-ns-check          skip checking if the hostname is resolvable.
--target,-t HOST1,HOST2  comma separated list of target hosts.
--timeout SEC            timeout for each command execution (defaults to 60).
--version,-v             print the version string and exit.
--mode,-m MODE           remote execution mode:
                         rsh - rsh based remote execution
                         nsh - nsh based remote execution
                         ssh - ssh based remote execution
                         local - execute the script locally, set TARGET_HOST on each execution

examples
========

1. run the command "ls" on host1 and host2.

$ nrun --target host1,host2 -- ls

2. run the command "ls -al" on host1, host2 and all hosts in the file HOSTS.LST.

$ nrun --target HOSTS.LST,host1,host2 -- ls -al

3. run the command "ls" on all hosts in the file HOSTS.LST - mode "ssh".

$ nrun --target HOSTS.LST --mode ssh -- ls 

4. copy the script bin/script.sh to each target hosts and execute it.

$ nrun --target HOSTS.LST --copy -- bin/script.sh

5. execute the local script test.sh for each target host

$ nrun --target HOSTS.LST --mode local -- ./test.sh

6. copy file test.tar to host1, host2 and all hosts in the file HOSTS.LST

$ ncopy --target HOSTS.LST -- test.tar /tmp 

configuration
=============

special configuration options for the different modes and additional all
commandline options can be given in a configuration file. the following three
places will be searched for configuration files (values in the last
configuration file will overwrite values in the first configuration file).
indentation does matter.

- $FindBin::Bin/../etc/nrunrc
- /etc/nrunrc
- $HOME/.nrunrc

--<snip>--
---
# mode ssh options
ssh_binary: /usr/bin/ssh
scp_binary: /usr/bin/scp
ssh_args:   -o PreferredAuthentications=publickey
scp_args:   -o PreferredAuthentications=publickey

# mode nsh options
agentinfo_binary: agentinfo
nexec_binary:     nexec
ncp_binary:       ncp
nexec_args:
agentinfo_args:
ncp_args:

# mode rsh options
rsh_binary: /usr/bin/rsh
rcp_binary: /usr/bin/rcp
rsh_user:   root
rcp_user:   root
rsh_args:
rcp_args:

# additional commandline options
arg_mode:     ssh
arg_parallel: 5
arg_timeout:  60

# alias definitions
alias:
    production:
        - host1
        - host2
--<snap>--

logging
=======

on each execution run, the command output and exit code will be saved inside the
logging directory. the default logging directory is $HOME/.nrun.

- $LOGDIR/results.log - will contain the exit codes
- $LOGDIR/output.log - will contain the complete command output for all hosts
- $LOGDIR/hosts/<HOSTNAME>.log - will contain the command output for a single
  host

mode ssh
========

use ssh as the underlying remote access mechanism.

the following configuration options must be set in the configuration file:

'ssh_args'   - arguments supplied to the ssh binary
'scp_args'   - arguments supplied to the scp binary
'ssh_binary' - ssh binary to be executed
'scp_binary' - scp binary to be executed

for passwordless login ssh-agent can be used:

# ssh-keygen
# scp .ssh/id_rsa.pub $USER@$HOST:.ssh/authorized_keys

# eval `ssh-agent` 
# ssh-add

to prevent any ssh interaction the following ssh command paramters are
suggested:

-o User=root
-o PreferredAuthentications=hostbased,publickey
-o StrictHostKeyChecking=no
-o UserKnownHostsFile=/dev/null
-o LogLevel=FATAL

mode local
==========

execute the script locally for each host and set the environment variable
TARGET_HOST on each execution.

mode nsh
========

use nsh as the underlying remote access mechanism.

the following configuration options must be set in the configuration file:

'agentinfo_args'   - arguments supplied to the agentinfo binary
'nexec_args'       - arguments supplied to the nexec binary
'ncp_args'         - arguments supplied to the ncp binary
'agentinfo_binary' - agentinfo binary to be executed
'nexec_binary'     - nexec binary to be executed
'ncp_binary'       - ncp binary to be executed

mode rsh
========

use rsh as the underlying remote access mechanism.

the following configuration options must be set in the configuration file:

'rsh_args'   - arguments supplied to the rsh binary
'rcp_args'   - arguments supplied to the rcp binary
'rsh_binary' - rsh binary to be executed
'rcp_binary' - rcp binary to be executed
'rsh_user'   - rsh login user
'rcp_user'   - rcp login user

inspecting long running processes

it is possible to signal nrun with USR1 and USR2 to dump, resp. save the currently
running processes. if nrun is signaled with USR2, it will create a file in
the log directory trace_<%Y%m%d_%H_%M_%S>.log.

the integer in square brackets is the pid of the perl process.

$ kill -USR1 1234
host1[14760]: (31800) /usr/bin/ssh -o User=root localhost sleep 120
host1[14760]: 1
host1[14760]: 2
host1[14760]: SIGUSR1 received
host1[14756]: (31801) /usr/bin/ssh -o User=root localhost sleep 120
host2[14756]: 1
host2[14756]: 2
host2[14756]: SIGUSR1 received

targets
=======

a target name may be either a filename containing the target hosts, one per line,
an alias definition in the configuration file or simply a hostname.

if there is a conflict, for example an alias named identically as an existing
file, the alias will always overrule the filename and the filename will always
overrule the hostname.

an alias can be defined in the configuration file the following way. an alias
definition may contain additional alias names, filenames or simply hostnames.
indentation does matter.

    # alias definitions
    alias:
        production:
            - host1
            - host2
        development:
            - host3
            - host4
        all:
            - production
            - development

transferring the public key
===========================

the helper script misc/put_pubkey can be used to transfer the ssh public key
to the target hosts without supplying a password for each login. it is meant
to be executed by the nrun script in mode local.

$ nrun -t HOSTS.LST --mode local --timeout 120 -- ./put_pubkey KEY USER PWD

a note on the git repository
============================

$ git clone https://github.com/tbenk/nrun
$ ln -s ../../.git_hooks/commit-msg .git/hooks

to enable keyword substitution:
$ git config --local filter.keywords.clean ".git_filters/keywords -m clean -f %f"
$ git config --local filter.keywords.smudge ".git_filters/keywords -m smudge -f %f"

to disable keyword substitution:
$ git config --local filter.keywords.smudge "cat"

after git keyword substitution is enabled, the following command will replace
all keywords in the source files:

$ git checkout HEAD .

for keyword substitution to work at least git version v1.7.3.4-599-ga2b665d is
necessary.

