# OpenInteract Administrator's Guide

This document goes over issues concerning system administrators, including mod_perl/Apache compilation and configuration, tips for running multiple websites under one mod_perl server, package configuration, logging, error reporting and caching.

While you're reading, it would be handy to have within reach the OpenInteract Glossary to lookup certain terms (like 'website', 'handler', 'package', etc.).

# oi_manage

Distributed with OpenInteract is the script `oi_manage`. If you installed OpenInteract via the normal Perl mechanisms, you should have it in your `/usr/bin` or `/usr/local/bin` directory so it's always accessible. (You might also ensure that `/usr/local/bin` is in your path and your users' paths, or that you create a symlink from `/usr/bin/oi_manage` to `/usr/local/bin/oi_manage`.)

This script takes care of many actions for you. Here's a list and a brief description of each:

    Commands by the Administrator:

     install             - Install OpenInteract to base directory
     upgrade             - Upgrade core OpenInteract packages
     install_package     - Install package to the base

    Commands by the Package Developer:

     create_skeleton     - Create a skeleton package for development
     export_package      - Export package(s) to distribution file(s)
     check_package       - Ensure that package passes initial inspection

    Commands by the Website Creator:

     create_website      - Create a new website
     apply_package       - Install a package from base to website
     upgrade_package     - Upgrade a website package
     remove_package      - Remove a package from a website
     install_sql         - Install the SQL for packages
     install_template    - Install package templates to the database
     dump_template       - Dump package templates to the filesystem
     remove_template     - Remove package templates from the database
     refresh_doc         - Sync website docs to base installation docs
     test_db             - Test database settings in 'server.perl'
     change_spops_driver - Change the SPOPS driver for your objects

    Other commands:

     initial_packages    - List packages marked as 'initial'
     list_packages       - List packages installed to app or base
     list_actions        - List actions currently implemented in website

You can read an online text version of the documentation for `oi_manage` For the most up-to-date information, do a `perldoc oi_manage` from the command line to view the extensive documentation.

# OpenInteract

This section reviews installation, layout and gives a very broad overview of the packaging system.

## Shortcuts

There are two environment variables you can use too make your typing life easier:

- `OPENINTERACT`: Base installation directory (used in place of `--base_dir`
- `OIWEBSITE`: Website directory (used in place of `--website_dir`

We encourage you to use these as much as possible. However, we continue to use the long form in the examples below.

## Installation

Installation of OpenInteract is very simple. First, you need to install the Perl module in the normal fashion. (CPAN or by-hand.) Next, run:

     cd /path/where/I/unpacked/OpenInteract-x.xx
     oi_manage install --base_dir=/path/to/install

Easy! Read the `INSTALL` and `INSTALL.website` files that were packaged with the OpenInteract distribution for more details.

## Layout

OpenInteract is composed of two pieces: the installation and the websites. The installation is what you create when you run the `oi_manage install` command above.

And you refer to it when you create new websites:

     oi_manage create_website --base_dir=/path/to/install \
                              --website_dir=/path/to/website \
                              --website_name=MySite

Thereafter, the website keeps the information for the base installation directory in its `conf/base.conf` file described below. So you shouldn't need to refer to the installation directory too often. (Developers need to know it when they are developing packages, but that's a separate issue.)

## Packages: Installing and Upgrading

So OpenInteract exists in two parts: the installation and the various websites that use the installation. When users who run the websites want to add new functionality by installing a new package or upgrading an existing one, they must first add the package to the OpenInteract installation and then apply it to the website.

The OpenInteract installation maintains a copy of every version of every package ever installed to the system. This might seem backwards or unnecessary, but it ensures that different websites can use and continue to use different versions of the same packages. This should allow you to accommodate different types of users: bleeding-edge developers who always play with new versions of packages and stay-the-line website maintainers who just want everything to work the same day after day.

Fortunately, the `oi_manage` script makes this whole process easy for us:

Install a new package to the installation:

     oi_manage install_package --package_file=/path/to/file.tar.gz \
                               --base_dir=/path/to/install

Install a new version of an existing package to the installation:

     oi_manage install_package --package_file=/path/to/file.tar.gz \
                               --base_dir=/path/to/install

Hey, those two were the same thing! It's true. Since the installation maintains a copy of every version of every package, it doesn't actually 'upgrade' a package in the installation.

This just emphasizes the main point: when you install a package to the OpenInteract installation, you're simply making it available for the websites that are using this installation.

Apply an installed package to a website:

     oi_manage apply_package --package=newpackage \
                             --website_dir=/path/to/website

Upgrade to a new version installed package for a website:

     oi_manage upgrade_package --package=newpackage \
                               --website_dir=/path/to/website

Ah, now those two commands were different. This shows that a website can only have one version of a particular package installed. If you try to `apply_package` for a package that has already been applied to a website, you'll get an error and no action will be taken. In this case you need to use `upgrade_package`.

The docs for `oi_manage` discuss this next point but it's worth repeating: when you use `upgrade_package`, you merely update the registry information for the website. You also install new data and structure, object configuration, templates, handlers and everything else.

However, the system does not remove the old files. The system does not know if you want to keep the changes you've made or discard them, so it simply keeps the old directory around and allows you to copy information back and forth as you need. But this old directory is not used by the system anymore, and if you make changes to files in the old package directory they will not be reflected in the website.

# Configuration File Formats

We generally use one of three formats. If you create new configuration files, try to stick to these naming schemes.

.conf

Typical configuration file format: information separated into key-value pairs (separated by whitespace), blank lines and lines that begin with a comment (#) are skipped.

Example:

      MyValue              emmet otter
      HerValue   fraggle rock
      TheirValue  jughandle
       # ringo   starr

Parsing this would return a hashref:

     {
       MyValue    => 'emmet otter',
       HerValue   => 'fraggle rock',
       TheirValue => 'jughandle'
     }

.dat

Very simple: one item per line. Blank lines and lines beginning with a comment (#) are skipped entirely.

Example:

     MyClass
     HerClass
     TheirClass
     #RingoClass

Parsing this would return an arrayref:

      [ 'MyClass', 'HerClass', 'TheirClass' ]

.perl

This file is a full-fledged perl data structure, dumped to a file using `Data::Dumper`. It can be any type of structure, but it's normally used to represent a hashref containing all sorts of different types of information. It's also fairly easy to edit such a file using your favorite plain-text file editor.

When reading this type of configuration, we just return the data structure saved in the file -- if the file is an arrayref, we return an arrayref.

When we use this structure to save information for objects (such as the `OpenInteract::Config::PerlFile` object), we never save class information about the object, just the data. We can always re-bless the structure after it's eval'd in.

Example:

     $data = {
              'db_info' => {
                'db_owner'    => '',
                'username'    => 'test',
                'password'    => '',
                'dsn'         => 'mydb',
                'db_name'     => 'myopeninteract',
                'driver_name' => 'mysql',
              },
              ...
     };

# Website Configuration

All files referenced in this section are contained in the home directory of the website. For instance, if you install a website using:

     oi_manage create_website --base_dir=/path/to/install \
                              --website_dir=/path/to/website \
                              --website_name=MySite

The files described here would be in `/path/to/website`.

We start with the simple files first and then get into the main configuration file.

## File: conf/base.conf

This is one of the most important configuration files in OpenInteract -- fortunately, nobody should never need to edit it. `:-)` This file allows `OpenInteract::Startup` to bootstrap all the configuration information by supplying information for the class that reads in the data for the server configuration (`OpenInteract::Config::PerlFile` by default), the base directory for this application, the name of the configuration directory and file. You can also specify the Request class (`OpenInteract::Request` by default) and a Stash class (no default specified -- every application needs to have its own).

Example:

    base_dir         /opt/OpenInteract
    website_dir      /home/httpd/mysite.com
    website_name     MySite
    config_type      perl
    config_class     OpenInteract::Config::PerlFile
    config_dir       conf
    config_file      server.perl
    request_class    OpenInteract::Request
    stash_class      MySite::Stash

## File: conf/apache.dat

List the classes needed by mod_perl in the `Apache::` class. You should never need to change this.

## File: conf/package_repository.perl

This is not a configuration file but included here for completeness. It is a text-based file (a Perl data structure in `Data::Dumper` format) and includes information about the packages in the website. You should never need to edit this file, but it's in a text-based format just in case you do.

## File: conf/server.perl

This is the main website configuration file and you may find yourself editing this quite often in the beginning and then every once in a while thereafter. The file is quite well-commented so you have the information about what/how to configure next to the actual data to change.

When you create a new website using `oi_manage` you get a starter `server.perl` file with a number of items filled in. (View the sample server.perl file -- note that keys surrounded with '%%' will be replaced when you create a website with `oi_manage`.

This configuration is always available within the OpenInteract environment (as an `OpenInteract::Config` object) to developers and is read in anew every time the server starts. If you change the file while the server is running the changes are not picked up by the running server. You need to restart to make the change happen. (This might change as the need arises.)

The file itself is formatted in Perl and you can edit it with your favorite plain-text editor. You should be able to check whether it is syntactically valid by doing:

     perl -wc server.perl

This won't tell you whether it's functionally valid, but it's a first step. Note that there is a routine in `oi_manage` that allows you to check whether the parameters defined in `server.perl` will allow a database connection to be made:

     oi_manage test_db --website_dir=/path/to/my/website

### Website Directories

The actual names of all directories are determined by this website configuration file as well. Here is an example with some commonly-used directories:

         'dir' => {
           'base'     => undef, # replaced in OpenInteract::Startup
           'interact' => undef, # replaced in OpenInteract::Startup
           'error'    => '$BASE/error',
           'html'     => '$BASE/html',
           'log'      => '$BASE/logs',
           'cache'    => '$BASE/cache',
           'config'   => '$BASE/conf',
           'data'     => '$BASE/data',
           'mail'     => '$BASE/mail',
           'overflow' => '$BASE/overflow',
           'help'     => '$HTML/help',
           'download' => '$HTML/downloads',
           'upload'   => '$BASE/uploads',
         },

Note that the 'dir' entry is one of the few places in the configuration where you can use expandable macros to determine the configuration value. As you might guess '$HTML' expands to find what the entry for 'html' is, which then depends on the value for 'base' is. You are not restricted to using the '$HTML' and '$BASE' macros.

Also, you can refer to '$BASE' even though the entry for 'base' is undefined. The standard `startup.pl` reads the value for 'website_dir' set in the httpd.conf and sets it dynamically once the config file is read in. This way you should only have to set your directory in one location. (And we should fix the slightly confusing naming scheme as well...)

Below are brief descriptions of what each directory should hold.

- base: The root directory for this application. By default, everything needed for this application is under this directory. (This is set dynamically in the `startup.pl` file so anything you set in the configuration file will be overwritten.)
- interact: The base installation directory. (This is set in the `startup.pl` file so anything you set in the configuration file will be overwritten.)
- error: If OpenInteract cannot save an error to the database, it will save the error (in `Data::Dumper` format) to this directory, using the ID of the error as a filename. Note that the error browser does not currently view these items, but this may change in the future.
- html: The root directory for static web pages and images.
- log: Directory where logs are kept. This needs to be writeable by the webserver user.
- cache: If you're using the default caching module that utilizes the filesystem, the caching will be done in this directory. This needs to be writeable by the webserver user.
- config: Where all the configuration information is kept; this includes SPOPS class definitions, module configuration, and the main configuration file itself. The C<`startup.pl`> file is also kept here.
- data: Not typically used in webserver operation, but usually where any data are kept, usually in a raw (CSV, bcp, etc.) format before being massaged and put into the database. Also used for pre-formed SQL statements to load data into the database. If a package you download has default data it needs to load, those data can be kept here.
- mail: Some applications may use asynchronous email services, where the system requests an email be sent but actually a file is written out with the necessary information and a time batch job (via cron) processes the mail at regular intervals. The files written for the batch job are written to this directory. This directory must be writeable by the webserver user.
- overflow: Often a search may be run that returns more information than is desired to fit on the screen. You can use this directory to hold intermediate results and retrieve them later. Note that this might be eliminated in favor of caching at a later date. This directory must be writeable by the webserver user.
- help: Directory where help files may be kept.
- download: Directory to put generated files that users need to access. For instance, you might generate a CSV and DBF file to hold exported search results, put the files here and offer the user links to this directory to download them. This directory must be writeable by the webserver user.
- upload: Directory where we can put uploaded files. This directory should not be in the HTML tree but it must be writeable by the webserver user.

## Aliases per Website

As a convenience, OpenInteract allows you to setup aliases so you do not have to put all sorts of class names in your code. For instance, instead of using a hardcoded classname:

     my $user = eval { OpenInteract::User->fetch( $uid ) };

You can use:

     my $user = eval { $R->user->fetch( $uid ) };

These aliases are setup automatically from the SPOPS classes, and you're also given the opportunity to setup any additional aliases using the 'system_aliases' configuration key. Here's an example:

     'system_alias' => {
       'OpenInteract::Cookies::Apache'    => [ qw/ cookies / ],
       'OpenInteract::Session::MySQL'     => [ qw/ session / ],
       'OpenInteract::Template::Toolkit'  => [ qw/ template / ],
       'OpenInteract::PackageRepository'  => [ qw/ repository / ],
       'OpenInteract::Package'            => [ qw/ package / ],
       'OpenInteract::Error'              => [ qw/ error / ],
       'OpenInteract::Auth'               => [ qw/ auth auth_user auth_group / ],
       '%%WEBSITE_NAME%%::Security'       => [ qw/ security_object object_security security / ],
       'SPOPS::Secure'                    => [ qw/ secure / ],
       'OpenInteract::Error::Main'        => [ qw/ error_handler / ],
       'OpenInteract::Handler::Component' => [ qw/ component / ],
     },

As you can see, you can setup multiple aliases to refer to the same class. For instance, any of the following will return the class 'OpenInteract::Auth':

     $R->auth
     $R->auth_user
     $R->auth_group

Note that using aliases like this is an as-yet-unmeasured performance hit, because every time you call the alias, $R needs to determine which application you're using so it can return the correct alias. This is a manifestation of the problem with running more than one application on the same server: what happens if two entirely separate applications want to setup an alias for 'news'?

You should never need to setup the alises by hand -- the standard `startup.pl` file will take care of this for you. However, if you're curious... You can setup the alises by calling:

     my $hr_alias = OpenInteract::Request->ALIAS;

Which returns a hashref to you. Any changes you make will be reflected in the master list of aliases. The hashref is setup as follows:

     $hr->{ $alias }->{ $stash_class } = $aliased_class;

So when you're setting up the aliases, you need to be sure and include your $stash_class in the alias, or else you'll get all kinds of errors.

Finally, we push the task of initializing the aliases (translating them into subroutines so they will execute quickly) into the ChildInitHandler, so we can ensure that all the applications have had their chance to modify the alias information before we create the subroutines.

## Website Logging

All debugging messages generated by the OpenInteract system are put into STDERR, which in apache is sent to the error log. You can always get the debugging level of the current website by running:

     my $website_debug_level = $R->DEBUG;

A common logging idiom, using the Cscrib()> method is:

      $R->DEBUG && $R->scrib( 1, "My debugging message" );

This first checks to see if debugging is on and only then calls the C method.

SPOPS modules do not obey this debugging level, and if you want to get debugging information from them you need to set the package variable `$SPOPS::DEBUG`. This turns on debugging for all SPOPS calls across websites.

# Caching

OpenInteract comes hooks for caching. While the caching framework is fairly generic (you could cache data in a SQL database or DBM file if you wished), the default caching mechanism is uses the `File::Cache` module and the filesystem. The module is fairly good about keeping the cache under a certain size (set in the configuration file), and you can also control the 'depth', which means that it creates subdirectories for the cache entries rather than putting them all into one big directory. Again, this is set in the configuration file. It's worth experimenting with this function as it works on your filesystem: journaling filesystems might be able to handle many thousands of entries in a single directory, whereas others might start having performance degradations after more than 500.

Most (all?) modern filesystems will put frequently accessed files into an in-memory cache, which can be a big win for caching. However, if yours does not one alternative is to put the caching directory itself in memory by using a RAM disk. Setting up such a filesystem is beyond the scope of this document.

# Apache

NOTE: DO NOT restart the Apache/mod_perl process using the `HUP` signal. Your modules will not get reloaded properly.

## Proxy Setup

OpenInteract depends on a persistent Perl environment within a web server. Currently, the best alternative is mod_perl.

mod_perl is extremely powerful, but this power can come at a price. Embedding Perl into Apache uses more resources (particularly memory) than just using Apache alone. A number of developers have experimented with various ways of minimizing the memory footprint of mod_perl, and one of the easiest and best performing methods is to use a proxy server.

This is described in great detail in the mod_perl guide under the Choosing the Right Strategy heading. But we'll summarize here:

1. Setup a plain Apache server with mod_proxy and mod_rewrite to listen to port 80 for your website. (We describe the build process below.)
2. Tell this server to deal with static file requests (images, movies, PDFs, etc.)
3. Proxy all other requests back to a heavier mod_perl server.
4. Receive the information back from the mod_perl server and send to the client.

The benefits of this are:

1. Resource-hogging mod_perl processes do not serve static files -- if they did, you'd need more of the processes.
2. The front-end proxy is able to feed data back to the client at whatever rate it needs without taking up many resources the entire time. For instance, users reaching your website with modems can tie up a web server process for much longer than users who are on some sort of broadband network. If the process is small it's not such a big deal.
3. Since they are separate, you can make changes to the (heavy) back-end and mask them by the (light) front-end. This is a great help when things are going wrong with the back-end and you don't want users to see nasty error pages.
4. Also since they are separate, you can very easily move the back-end process to an entirely separate machine (or machines, using some sort of DNS or load-balancing manipulation) if the need arises.

Running OpenInteract in this environment is strongly recommended, and it comes with configuration files that make it easier to do the Right Thing.

## Building Apache: Proxy and mod_perl

First, you need to get the `mod_proxy_add_forward` module make available by Ask Bjoern Hansen. Retrieve it from: http://develooper.com/code/mpaf/mod_proxy_add_forward.c

Once you've retrieved the file, copy it into the `src/modules/extra` directory of your Apache source code directory. An example of the 'activate-module' and 'enable-module' directives to put this module into your Apache is below in the as well as in the source code for `mod_proxy_add_forward` itself.

Once you've retrieved the extra module and copied it to the right place, you can create apache and mod_perl with the following steps. Note that this assumes you have not installed apache from source before and that you're installing to the directory `/usr/local/apache` -- modify as needed.

     1.  >> tar -zxvf apache-1.3.12.tar.gz

     2.  >> tar -zxvf mod_perl-1.24.tar.gz

     3.  >> cd apache-1.3.12

     4.  >> ./configure --prefix=/usr/local/apache \
                    --enable-module=rewrite --enable-module=proxy \
                    --activate-module=src/modules/extra/mod_proxy_add_forward.c \
                    --enable-module=proxy_add_forward

     5.  >> make

     6.  >> make install
     (proxy server binary is now installed as /usr/local/apache/bin/httpd)

     7.  >> cd ../mod_perl-1.24

     8.  >> perl Makefile.PL EVERYTHING=1
     # Configure mod_perl with ../apache_1.3.12/src ? [y]

     9.  >> y
     # Shall I build httpd in ../apache_1.3.12/src for you? [y]

     10. >> y

     11. >> make

     12. >> make test

     (note: if this fails due to an error with URI::URL, set the
     environment variable 'PERL_HTTP_URI_CLASS' to 'URI::URL', with
     something like:

        # export PERL_HTTP_URI_CLASS=URI::URL

     13. >> make install
     (mod_perl Perl modules are now installed)

     14. >> cp ../apache-1.3.12/src/httpd /usr/local/apache/bin/httpd_modperl
     (mod_perl-enabled Apache is now installed)

This is a very simple method for creating both a lightweight proxy Apache binary and a heavyweight mod_perl-enabled Apache binary. See the mod_perl Guide for many, many more details about building mod_perl.

It is strongly recommended that you do not build mod_perl using DSOs and that you do not use pre-built versions such as those supplied by RedHat with its RPMs. However, using the DSO mechanism probably works fine for the front-end proxy server.

## Configuration Overview

Use `oi_manage`! Use `OI_MANAGE`! Use `OI_MANAGE`!

The `oi_manage` script included with OpenInteract performs a number of tasks for you that make your life much easier. When you run the `create_website` command along with the appropriate parameters, `oi_manage` will copy configuration files from the base installation to your website directory and customize them for your website's parameters for you.

For instance, two of the files that are copied to your website's `conf/` directory are `httpd_static.conf` and `httpd_modperl.conf`. (See httpd_static.conf and httpd_modperl.conf -- the items marked with '%%' are replaced in the customization process.) You will still need to edit a few parameters in them -- `oi_manage` is pretty smart, but it can't find out which IP address you want your website to listen to! -- but much of it is filled in for you.

## Static Apache Configuration

After you've run `oi_manage`, you will need to modify a few parameters in the static Apache configuration file.

1. IP address: Do a search-replace for '127.0.0.1' with the IP address you want the website to listen to. Note that if you're using named virtual hosts you will not want to keep the `Listen` directive. You will also need to specify the `NameVirtualHost` directive in your main Apache configuration file.
2. ServerAdmin: Change the value for the 'ServerAdmin' key
3. ServerName: Change the value for the 'ServerName' key

Proxy configuration is fairly simple. Every rule (starting with `RewriteRule`) is processed in order. Once a rule is met, no further rules are processed unless the satisfied rule specifies it.

The default proxy configuration assumes that the only static files you will want to serve directly from the proxy server are images. This action is specified by this line:

     RewriteRule ^/images - [L]

If you want to add other locations that will be entirely served by the lightweight server, just add them after this line. For example, if my website had a directory '/forms' where we kept PDF versions of forms for our customers to fill out, I could add:

     RewriteRule ^/forms - [L]

And every URL beginning with `/forms` will be answered by the front-end lightweight server. The `[L]` stands for "Local" and means that you want this server (the proxy server) to handle the request.

The only word of warning here is that as an administrator you might need to keep an eye on what the back-end server is using for URLs. For instance, say I entered this `/forms` configuration directive and later a developer on the back-end server tries to configure OpenInteract to perform a certain action when given the `/forms` URL. Unless the developer knows that the front-end server is answering all the `/forms` URLs, she will have a very frustrating time trying to figure out why her handler isn't responding.

## mod_perl Configuration

After you've run `oi_manage`, you will need to modify a few parameters in the mod_perl Apache configuration file.

1. IP address: Do a search-replace for '127.0.0.1' with the IP address you want the website to listen to.
2. ServerAdmin: Change the value for the 'ServerAdmin' key
3. ServerName: Change the value for the 'ServerName' key
4. Port: (optional) Do a search-replace for the default value of '8080' with whatever port you want to run the mod_perl server on

(Note: You can skip the remainder of this section if you just want to get something up and running. The `oi_manage` script takes care of all this for you. But if you're curious, read on.)

Four separate items need to be customized in the `conf/httpd_modperl.conf` file:

First, define the library paths for this website. Note that this is applied on a server-wide basis, so be careful of namespace clashes.

Example:

     <Perl>
      use lib qw( /home/httpd/mysite.com );
     </Perl>

Second, define a parameter that allow us to bootstrap our configuration object which contains the rest of everything we need to know. The parameter is 'StashClass' you set it to a value that OpenInteract can use for a website's stash class.

Example:

     PerlSetVar StashClass      MySite::Stash

Third, you need to bring in your `startup.pl`. (Information on what is done in the `startup.pl` is found in the OpenInteract Developer's Guide.)

     PerlRequire /home/httpd/mysite.com/conf/startup.pl

Fourth and finally, we need to ensure that every request coming in goes through a single Apache content handler: `OpenInteract.pm`. (This module is located in the `base` package.) To enable this, just do:

     <Location />
      SetHandler perl-script
      PerlHandler OpenInteract
     </Location>

This Apache content handler is in the `base` package since it's part of the base functionality of the framework. We can just say "OpenInteract" in the httpd.conf file because we have already included the library in our `startup.pl`.

## Running Multiple Websites on One Server

OpenInteract is a fairly heavyweight application. Its numerous modules and, more importantly, package versioning necessitate that each application run under its own mod_perl server. Multiple applications can easily be run on the same machine, and can share the same base package repository. But experience has shown that trying to run multiple applications under the same set of processes is nearly impossible unless everything is kept perfectly in sync. And that never, ever happens.

# Suggested Readings

- mod_perl Guide
  http://perl.apache.org/guide/
- General Apache documentation
  http://www.apache.org/docs/
- mod_rewrite manual
  http://www.apache.org/docs/mod/mod_rewrite.html
- Apache Virtual Host documentation
  http://www.apache.org/docs/vhosts/index.html