manpagesection(1. SPECIFICATION FILE(S))

    Flc() expects an input file containing directives and the regular
expressions that should be recognized by objects of the scanner class
generated by flc(). In this man page the elements and organization of flc()'s
input file is described. 

    Flc()'s input file consists of two sections, separated from each other by
a line merely containing two consecutive percent characters:
    verb(
%%
    )
    The section before this separator contains directives; the section
following this separator contains regular expressions and possibly actions to
perform when these regular expressions are matched by the object of the
scanner class generated by flc(). If a second line is encountered immediately
beginning with  two consecutive percent characters then this ends flc()'s
input file processing. See also section 6 (%% SEPARATOR) below.

    White space is usually ignored, as is comment, which may be of the
traditional bf(C) form (i.e., tt(/*), followed by (possibly multi-line)
comment text, followed by tt(*/), and it may be bf(C++) end-of-line comment:
two consecutive slashes (tt(//)) start the comment, which continues up to
the next newline character.

manpagesection(2. FILE SWITCHING)

Flc()'s input file may be split into multiple files. This allows for the
definition of logically separate elements of the specifications in different
files. Include directives must be specified on separate lines and may not
contain any other information than the (path)name of the file to switch
to. File names may be surrounded by double quotes, but these double quotes are
optional and are ignored (removed) when encountered. All remaining characters
define the name of the subsequently processed file by flc().  White space
characters following tt(//include) and preceding the end of the line are
ignored. To switch files the following stanza is used:
        verb(
//include file-location
        )
    The tt(//include) directive must start in the line's first column. File
locations can be absolute or relative to the location of the file containing
the tt(//include) directive. Once flc() has switched to another file that
file's directory becomes flc()'s working directory.

Once the end of file of a file has been reached, processing continues at
the line beyond the tt(//include) directive of the previously scanned file,
and flc()'s working directory is reset to the working directory of the file to
which flc() returns. The end-of-file of the file that was initially specified
when flc() started indicates the end of flc()'s rules specification.

manpagesection(3. DIRECTIVES)

    The first section of flc()'s input file consists of directives. In
addition it may associate regular expressions with symbolic names, allowing
you to use these identifiers in the rules section. Each directive is defined
on a line of its own. When available, directives are overridden by flc()
command line options.

    includefile(../manual/lexer/directives.yo)

manpagesection(4. MINI SCANNERS)

    Mini scanners come in two flavors: inclusive mini scanners and exclusive
mini scanners. The rules that apply to an inclusive mini scanner are the mini
scanner's own rules as well as the rules which apply to no mini scanners in
particular (i.e., the rules that apply to the default (or tt(INITIAL)) mini
scanner). Exclusive mini scanners only use the rules that were defined for
them. 

    To define an inclusive mini scanner use tt(%s), followed by one
or more identifiers specifying the name(s) of the mini-scanner(s). To define
an exclusive mini scanner use tt(%x), followed by or more identifiers
specifying the name(s) of the mini-scanner(s). The following example defines
the names of two mini scanners: tt(string) and tt(comment): 
        verb(
    %x string comment 
        )
    Following this, rules defined in the context of the tt(string) mini
scanner (see below) will only be used when that mini scanner is active.

    A flc() input file may contain multiple tt(%s) and tt(%x)
specifications.

manpagesection(5. DEFINITIONS)

    Definitions are of the form
        verb(
identifier  regular-expression
        )
    Each definition must be entered on a line of its own. Definitions
associate identifiers with regular expressions, allowing the use of
tt(${identifier}) as synonym for its regular expression in the rules section
of flc()'s input file. One defined, the identifiers representing regular
expressions can also be used in subsequent definitions.

Example:
        verb(
FIRST                   [A-Za-z_]
NAME                    {FIRST}[-A-Za-z0-9_]*
        )

manpagesection(6. %% SEPARATOR)

Following directives and definitions a line merely containing two consecutive
tt(%) characters is expected. Following this line the rules are defined. Rules
consist of regular expressions which should be recognized, possibly followed
by actions to be executed once a rule's regular expression has been matched.

If the rule section contains a line starting with two consecutive tt(%)
characters, then any remaining input is ignored. Note that this second tt(%%)
separator does not have to be specified. It is purely optional. To specify a
regular expression starting with tt(%%) surround the tt(%%) with double quotes
(tt("%%")) or prefix the tt(%%) with a blank space: the tt(%%)-characters are
only considered a separator if they are encountered at the very beginning of a
line. 

manpagesection(7. REGULAR EXPRESSIONS)

The regular expressions defined in flc()'s rules files are matched against 
the information passed to the scanner's tt(lex) function. 

Regular expressions begin as the first non-blank character on a line. Comment
is interpreted as comment as long as it isn't part of the regular
expresssion. To define a  regular expression starting with two slashes (at
least) the first slash can be escaped or double quoted. (E.g., tt("//".*)
defines bf(C++) comment to end-of-line).

Regular expressions end at the first blank character (to add a blank character,
e.g., a space character, to a regular expression, prefix it by a backslash or
put it in a double-quoted string).

Actions may be associated with regular expressions.  At a match the action
that is associated with the regular expression is executed, after which
scanning continues when the lexical scanning function (e.g., tt(lex)) is
called again. Actions are not required, and regular expressions can be defined
without any actions at all. If such action-less regular expressions are
matched then the match is performed silently, after which processing
continues.

Flc() tries to match as many characters of the input file as possible (i.e.,
it uses `greedy matching'). Non-greedy matching is accomplished by a
combination of a scanner and parser and/or by using the `lookahead' operator
(tt(/)).

The following regular expression `building blocks' are available. More complex
regular expressions are created by combining them:

description(
dit(tt(x))
    the character `x';
dit(tt(.))
    any character (byte) except newline;
dit(tt([xyz]))
   a character class; in this case, the pattern matches either an `x',
    a `y', or a `z'. See also the paragraph about character classes below;
dit(tt([abj-oZ]))
   a character class containing a range; matches an `a', a `b', any
    letter from `j' through `o', or a `Z'. See also the paragraph about
    character classes below;
dit(tt([^A-Z])) a negated character class, i.e., any character except
    for those in the class.  In this example, any non-capital character. See
    also the paragraph about character classes below; 
dit(tt("[xyz]\"foo"))
    text between double quotes matches the literal string: tt([xyz]"foo);
dit(R"([xyz]\"foo)")
    the literal string  `tt([xyz]\"foo)' (using a raw string literal);
dit(tt(\X))
   if X is `a', `b', `f', `n', `r', `t', or `v', then the ANSI-C
    interpretation of `\x' is matched. Otherwise, a literal `X' is matched
    (this is used to escape operators such as `*');
dit(tt(\0))
    a NUL character (ASCII code 0);
dit(tt(\123))
    the character with octal value 123;
dit(tt(\x2a))
    the character with hexadecimal value 2a;
dit(tt((r)))
   the regular expression `r'; parentheses are used to override
    precedence (see below);
dit(tt({name}))
   the expansion of the `name' definition;
dit(tt(r*)) 
   zero or more regular expressions `r'. This also matches the empty
    string;
dit(tt(r+))
   one or more regular expressions `r';
dit(tt(r?))
   zero or one regular expression `r'.  This also matches the empty
    string;
dit(tt(rs))
   the regular expression `r' followed by the regular expression `s';
    called concatenation;
dit(tt(r{m, n})) 
   regular expression `r' at least m, but at most n times (tt(0 <= m
    <= n)).  A regular expression to which tt({0, 0}) is appended
    is ignored, and a warning message is shown.
dit(tt(r{m,}))
   regular expression `r' m or more times (tt(0 <= m));
dit(tt(r{m}))
   regular expression `r' exactly m times (tt(0 <= m)).  A regular expression
    to which tt({0}) is appended is ignored, and a warning message is shown; 
dit(tt(r|s))
   either regular expression `r' or regular expression `s';
dit(tt(r/s))
   regular expression `r' if it is followed by regular expression
    `s'. The text matched by `s' is included when determining whether this
    rule results in the longest match, but `s' is then returned to the input
    before the rule's action (if defined) is executed.

   If flc() detects patterns potentially not matching any text it generates 
    warnings like this:
        verb(
    [Warning] input, line 7: null-matching regular expression
        )
    By placing the comment
        verb(
    //%nowarn
        )
   on the line just before a regular expression that potentially does not
    match any text, the warning for that regular expression is suppressed;

dit(tt(^r))
   a regular expression `r' at the beginning of a line or file;
dit(tt(r$))
   a regular expression `r', occurring  at the end of a line. This
    pattern is identical to `r/\n';
dit(tt(<s>r))
   a regular expression `r' in start condition `s';
   (don't use white-space characters between the tt(<) and tt(>) characters
    around start condition(s));
dit(tt(<s1,s2,s3>r))
   a regular expression `r' in start conditions s1, s2, or s3+nl()
dit(tt(<*>r))
   a regular expression `r' in all start conditions;
dit(tt(<)tt(<EOF>)tt(>))
   an end-of-file;
dit(tt(<s1,s2><)tt(<EOF>)tt(>))
   an end-of-file when in start conditions s1 or s2 .
)

bf(Character classes)

Inside a character class all regular expression operators lose their special
meanings, except for the escape character (tt(\)), the character range
operator tt(-), the end of character class operator tt(]), and, at the
beginning of the class, tt(^). All ordinary escape sequences are supported,
all other escaped characters are interpreted as literal characters (e.g.,
tt(\c) is a literal tt(c)).

To add a closing bracket to a character class use tt([]) or tt(\]). To add a
closing bracket to a negated character class use tt([^]) (or use tt([^)
followed by tt(\]) somewhere within the character class). Minus characters are
used to define character ranges (e.g., tt([a-d]), defining tt([abcd])) except
in the following cases, where flc() recognizes a literal minus character:
    tt([-), or tt([^-) (a minus at the very beginning of a character class); 
    tt(-]) (a minus at the very end of a character class); 
    or tt(\-) (an escaped minus character))
 Once a character class has started, all
subsequent character (ranges) are added to the set, until the final closing
bracket (tt(])) has been reached.

bf(Character constants)

includefile(../charconsts.yo)

bf(Operator precedence)

The regular expressions listed above are grouped according to precedence, from
highest precedence at the top to lowest at the bottom. From lowest to highest
precedence, the operators are:
        itemization(
        itt(|): the or-operator at the end of a line (instead of an action)
indicates that this expression's action is identical to the action of the next
rule. 
        itt(/): the look-ahead operator;
        itt(|): the or-operator withn a regular expression;
        itt(CHAR): individual elements of the regular expression: characters,
strings, quoted characters, escaped characters, character sets etc. are all
considered tt(CHAR) elements. Multiple tt(CHAR) elements can be combined by
enclosing them in parentheses (e.g., tt((abc)+) indicates sequences of tt(abc)
characters, like tt(abcabcabc));
        itt(*, ?, +, {): multipliers:nl()
            tt(?): zero or one occurrence  of the previous element;nl()
            tt(+): one or more repetitions of the previous element;nl()
            tt(*): zero or more repetitions of the previous element;nl()
            tt({...}): interval specification: a specified number of
                repetitions of the previous element (see above for specific
                forms of the interval specification)
        itt({+}, {-}): set operators (tt({+}) computing the union of two sets,
                tt({-}) computing the difference of the left-hand side set
                minus the elements in the right-hand side set);
        )

The lex standard defines concatenation as having a higher precedence than the
interval expression. This is different from many other regular expression
engines, and flc() follows these latter engines, giving all `multiplication
operators' equal priority.

Name expansion has the same precedence as grouping (using parentheses to
influence the precedence of the other operators in the regular expression).
Since the name expansion is treated as a group in flc(), it is not allowed to
use the lookahead operator in a name definition (a named pattern, defined in
the definition section).

bf(Predefined sets of characters)

Character classes can also contain character class expressions. These are
expressions enclosed inside tt([:) and tt(:]) delimiters (which themselves
must appear between the tt([) and tt(]) of the character class. Other elements
may occur inside the character class as well). The character class expressions
are:
        verb(     
     [:alnum:] [:alpha:] [:blank:]
     [:cntrl:] [:digit:] [:graph:]
     [:lower:] [:print:] [:punct:]
     [:space:] [:upper:] [:xdigit:]
        )

    Character class expressions designate a set of characters equivalent to
the corresponding standard bf(C) isXXX function. For example, tt([:alnum:])
designates those characters for which tt(isalnum) returns true - i.e., any
alphabetic or numeric character.  For example, the following character classes
are all equivalent:
        verb( 
    [[:alnum:]]
    [[:alpha:][:digit:]]
    [[:alpha:][0-9]]
    [a-zA-Z0-9]
        )

A negated character class such as the example tt([^A-Z]) above will match a
newline unless tt(\n) (or an equivalent escape sequence) is one of the
characters explicitly present in the negated character class (e.g.,
tt([^A-Z\n])). This differs from the way many other regular expression tools
treat negated character classes, but unfortunately the inconsistency is
historically entrenched. Matching newlines means that a pattern like tt([^"]*)
can match the entire input unless there's another quote in the input.

Flc() allows negation of character class expressions by prepending tt(^) to
the POSIX character class name.
        verb(                
    [:^alnum:] [:^alpha:] [:^blank:]
    [:^cntrl:] [:^digit:] [:^graph:]
    [:^lower:] [:^print:] [:^punct:]
    [:^space:] [:^upper:] [:^xdigit:]
        )      

bf(Combining character sets)

The tt({-}) operator computes the difference of two character classes. For
example, tt([a-c]{-}[b-z]) represents all the characters in the class
tt([a-c]) that are not in the class tt([b-z]) (which in this case, is just the
single character tt(a)). The tt({-}) operator is left associative, so
tt([abc]{-}[b]{-}[c]) is the same as tt([a]).

The tt({+}) operator computes the union of two character classes. For example,
tt([a-z]{+}[0-9]) is the same as tt([a-z0-9]). This operator is useful when
preceded by the result of a difference operation, as in,
tt([[:alpha:]]{-}[[:lower:]]{+}[q]), which is equivalent to tt([A-Zq]) in the
bf(C) locale.

bf(Trailing context)

A rule can have at most one instance of trailing context (the tt(/) operator
or the tt($) operator). The start condition, tt(^), and tt(<<EOF>>) patterns
can only occur at the beginning of a pattern, and cannot be surrounded by
parentheses. The characters tt(^) and tt($) only have their special properties
at, respectively, the beginning and end of regular expressions. In all other
cases they are treated as a normal characters.

manpagesection(8. SPECIFICATION EXAMPLE)

    verb(
%option debug

%x comment

NAME    [[:alpha:]][_[:alnum:]]*

%%

"//".*          // ignore

"/*"            begin(StartCondition_::comment);

<comment>.|\n   // ignore
<comment>"*/"   begin(StartCondition_::INITIAL);

^a              return 1;
a               return 2;
a$              return 3;
{NAME}          return 4;

.|\n            // ignore
        )

    )
