=head1 NOTE 

This document is probably no longer current.

=head1 PARSING

This document explains the algorithm used by C<parse.imc> to parse a 
hunk of tcl.
This is a from-scratch implementation based on the tcl man page. This was, btw,
a heck of a lot easier when I had perl5's regexps to do things with. =-)

First, in C<__main>, we read in the input file and shove it in a string. This
then gets passed to the C<__parse> sub. (Or we take stdin, we're not picky)

Footnotes refer to bullet items in the tcl man page.

=head1 STATE_MACHINE

There are several states that our parser can be in:

=head2 BEGIN_SCOPE

Where we begin, create a lexical scope in which to store variables.

newline/backslash substitution is performed on the string. [8]

=head2 BEGIN_COMMAND

We clear out the Array that is holding our command.

=head2 BEGIN_WORD

skip any leading whitespace. If a newline or a ; is found, goto END_COMMAND [1]

if the first character of the first word is a #, then it's a comment:
ignore all characters until the next newline, and go to BEGIN_COMMAND [9]

If the first character of a word is a double-quote, the word consists of
all the characters between the two double quotes. append it to the command
Array and goto BEGIN_WORD [4] (escaped \"'s are ignored.) 
(any non whitespace/
non command separator character in the stream at this point is an error.)

If the first character of a word is a {, the word consists of all the characters
between the { and the }. append the word to the command Array and goto begin word. [5] There must be a matched number of unescaped { and } chars.
(any non whitespace/
non command separator character in the stream at this point is an error.)

If there are no more characters, goto END_SCOPE

If any other character, then fall through:

=head2 MIDDLE_WORD

We're in the middle of getting a word. Any whitespace indicates END_WORD. A ;
or \n indicates END_COMMAND.

If a [ (unescaped) is present, then the word extends to at least the next ]. 
Grab these characters, goto MIDDLE_WORD

If a ${ appears, the word extends to at least the next }. Grab these characters,
goto MIDDLE_WORD

=head2 END_WORD

We've reached the end of a word. Add it to the array of words. goto BEGIN_WORD

=head2 END_COMMAND

We've reached the end of a command, append any outstanding word into the 
command array.

Append the command array to the array of commands. Goto BEGIN_COMMAND

=head2 END_SCOPE

We now have an array of arrays, which correspond to the raw text of the words
in the code. Now we need to perform various substitutions on the words. 
(In a future version, this is where we'd compile the code. For now, we'll
just interpret it.)

=head2 RUN_COMMAND

pop an array off the array of commands. For each of the words in the command
array, we need to make sure we only process each character of text once - to do
this, we keep a linked list of { state, start, len } - Each round of
substitution can only happen on raw segments. Once a substitution occurs, the
list is further segmented, the raw being broken up into possibly multiple
alternating raw/cooked segments. Substitutions are NOT done on words that
were {} words.

=over 4 

=item Command substitution

All characters between a [ and ] are considered a
tcl script, and run through a separate invocation of the parser. 

=item Variable substitution

If there's a $ , then any of the following text
is replaced with the corresponding variable value: $name , $name(index)
and ${name}. index has command, variable, and backslash
substitutions performed on it before it's used to lookup a value.

=item backslash substitution

Various \ substitutions, except for backslash-newline, which is done before
anything else when we first get our script.

=head2 EXECUTE_COMMAND

At this point, each of the words is as cooked as it's going to be. Put the
list for each word back together into a single string. Call the command
associated with the first cooked word and pass in the rest of the array
as the parameters.

Save return value. (but only the last one)

While there are commands left, go to RUN_COMMAND

return the last return value saved. (XXX: what to return if there was no 
command executed? empty string?)
