essence(1)

NAME
Essence - Customized information extraction and indexing

SYNOPSIS
essence [ -i ] [ -noindex ] [ -n index-name ] [ -s stoplist ] [ -t tmpdir ] [ -v ] ( -f files datadir | datadir directory ... )

DESCRIPTION

essence supports application-specific, user-customizable means of extracting and indexing information from a wide variety of file system data. By exploiting semantics of common file types, essence generates compact yet representative file summaries that can be indexed and exported via the Wide Area Information Servers (WAIS) search and retrieval interface. People who wish to index data and search it using WAIS can use essence to index more file types than WAIS itself currently supports, and to produce more space efficient indexes.

RUNNING ESSENCE

Run the essence command to generate file summaries and optionally a WAIS index for a portion of a file system. You must supply essence with the name of the directory, called datadir, in which the file summaries and the optional WAIS index will be placed. To specify which portion of the file system on which you want to run essence, you can list the top-level directory name(s) and essence will use all of the files and directories that they contain, or you can explicitly list the individual files and directories to use in a text file, giving essence the -f flag.

Refer to the "CUSTOMIZING ESSENCE" section (below) for information on modifying the heuristics that essence uses.

OPTIONS

-f files
    Specifies the file which lists the files and directories on which to run. The default is to get the top-level directories from the command line.

-i  Interactive mode. Before generating an index, essence will wait for the use to manually add any comments that they wish to the summary files. The default is not to use the interactive mode.

-noindex
    Specifies to only generate summary files. No index is produced. The default is to generate summary files and to produce an index.

-n index-name
    Specifies the name of the resulting WAIS index. ``Essence'' is the default WAIS index name.

-s stoplist
    Specifies the name of the stop list file. ``stoplist'' is the default name.

-t tmpdir
    Specifies the directory in which all temporary files will be placed. Note that essence may use many large temporary files when given nested files. ``/tmp'' is the default directory.

-v  Verbose mode.

EXAMPLE

To generate a WAIS index for an Anonymous FTP area in /ftp, use:
        essence /usr/local/essence/data /ftp

where /usr/local/essence/data is the directory in which the summaries and the WAIS index will be held. In this example, the name of the WAIS index will be the default "Essence".

To rename the WAIS index to ``aftp-myhost'', use
        essence -n aftp-myhost /usr/local/essence/data /ftp

To use an alternate temporary directory, use
        essence -t /extra/tmp /usr/local/essence/data /ftp

If you want to add comments to summary files before indexing them, use
        essence -i /usr/local/essence/data /ftp
After generating the summaries, essence will prompt you to add any comments that you want to them. essence generates the datadir/Map file to help you decide which summary files you'd like to edit.

CUSTOMIZING ESSENCE

essence uses four basic steps to extract information: classification, presentation unnesting, candidate selection, and summarizing. Each of these steps is user-customizable.

Classification

This prototype uses three methods for classifying files: file naming conventions, locating identifying data within a file, and external classification programs. Users can customize the mechanisms used for each of these methods, by modifying the .cf configuration files or writing external classification programs.

The filename.cf file contains all of the heuristics for classifying files through file naming conventions. Regular expressions are used to specify file naming heuristics. Then each heuristic is associated with a specific file type. For example, the regular expression '.*.Z' (any file ending in a '.Z') is associated with the 'Compressed' file type. with the 'PostScript' file type.

essence uses the file(1) command (included in the essence distribution) to locate identifying data within a file. The magic(5) file contains all of the heuristics for locating identifying data within a file. The fileout.cf file contains regular expressions that associate the output of the file command with specific file types. For example, if the file command outputs text that matches the regular expression essence classifies the file as type 'Image'.

Finally, users can write a program uses whatever means available to classify a file or directory. These classification programs take a single argument, the name of the file or directory to classify. Each program tries to classify a file or directory as a specific type. If it succeeds in classifying the file or directory as a specific type, then it exits with a return value of 0; otherwise, it returns a non-zero value. Each program is named after the type that it is trying to identify. If the program is trying to identify a type for a directory, then it uses the '.identd' suffix; otherwise, it uses the SourceDistribution.identd program is name of the classification program that tries to identify directories that contain a source distribution.

Presentation Unnesting

This prototype hard-codes all of the presentation unnesting heuristics in the engine program. To add a heuristic you need to modify the two procedures: init_nested_structure_type() and ns_processor().

Candidate Selection

This prototype only supports limited functionality for candidate selection. Users can elect to prevent essence from indexing any file of a specific file type by adding the name of the file type to the stoplist file.

Summarizing

Once an object has been classified, possibly unnested, and selected for indexing, then a summarizing program (called a summarizer) is used to extract the "important" information from the file or directory. Summarizers are programs that use whatever means available to extract information from objects to use in the summary files. Summarizers take a single argument, the name of the file or directory to summarize, and output whatever text that they wish to include in the summary file. Summarizers are named after the file type that they summarize and use the suffix '.sum'. For example, the PostScript.sum program is the summarizer for the PostScript file type.

WAIS INTERFACE

essence exports its indexes through WAIS's search and retrieval interface, allowing users to use tools such as waissearch and the X Windows-based graphical user interface xwais. To simplify generating WAIS-compatible indexes, the prototype uses WAIS's indexing software to build an index. In turn, essence simply runs waisindex on the generated summary files to build a full-text WAIS index.

Basically, we modified the WAIS indexing mechanism to understand the format of the essence summary files, so that it generates meaningful WAIS headlines. These headlines provide users with a short description of a single file, usually a filename. In essence, headlines represent a filename, its file type, and a file's core filename.

Since essence only supports the retrieval of summary files via WAIS, we've modified WAIS to always return the EssenceInterface file as the first hit. This file briefly describes how the WAIS interface to essence.

Prototype Availability

The essence prototype, including its source code and WAIS modifications, is publically available by anonymous FTP from ftp.cs.colorado.edu in /pub/cs/distribs/essence. The prototype is written in the C and Perl programming languages.

FILES

$datadir/Map
    Type and summary file information for each summarized file.

$datadir/sums
    Location of summary files.

SEE ALSO

file(1), perl(1), waisindex(1), waissearch(1), waisserver(1), xwais(1), and xwaisq(1).

``Essence: A Resource Discovery System Based on Semantic File Indexing,'' by D. R. Hardy and M. F. Schwartz, 1993 Winter USENIX Technical Conference Proceedings, San Diego, CA, January 1993, pp. 361-374. This publication discusses the design, performance, and measurements of Essence, and compares it with related systems. It is available via anonymous ftp from ftp.cs.colorado.edu as /pub/cs/techreports/schwartz/PostScript/Essence.ps.Z or /pub/cs/techreports/schwartz/ASCII/Essence.txt.Z.

AUTHORS

    Darren R. Hardy
    Department of Computer Science
    University of Colorado
    Boulder, CO  80309-0430
    hardy@cs.colorado.edu

    Michael F. Schwartz
    Department of Computer Science
    University of Colorado
    Boulder, CO  80309-0430
    schwartz@cs.colorado.edu

COPYRIGHT

Copyright (c) 1992, 1993, Darren R. Hardy and Michael F. Schwartz of the University of Colorado at Boulder. All rights reserved.

This software was developed as part of the Networked Resource Discovery Project. Users may copy, modify, and redistribute the code without charge, provided that the above copyright notice and this paragraph are duplicated in all such forms and that any documentation and other materials related to such distribution and use acknowledge that the software was developed at the University of Colorado at Boulder. This software may not be sold or used for profit.

This software is provided "as is", without express or implied warranty, and with no support nor obligation to assist in its use, correction, modification or enhancement. We assume no liability with respect to the infringement of copyrights, trade secrets, or any patents, and are not responsible for consequential damages. Proper use of this software is entirely the responsibility of the user.