XTOKID(1)

NAME
xtokid - Print all tokens found in a source file.

SYNOPSIS
xtokid [OPTION]... [FILE]...

DESCRIPTION

Print all tokens found in a source file.

-i, --include=LANGS
    include languages in LANGS (default: "C C++ asm")

-x, --exclude=LANGS
    exclude languages in LANGS

-l, --lang-option=L:OPT pass OPT as a default for language L (see below)

-m, --lang-map=MAPFILE
    use MAPFILE to map file names onto source language

-d, --default-lang=LANG
    make LANG the default source language

-p, --prune=NAMES
    exclude the named files and/or directories

--files0-from=F
    tokenize only the files specified by NUL-terminated names in file F

--help
    display this help and exit --version output version information and exit

The following arguments apply to the language-specific scanners:

  C language:

-k,--keep=CHARS
    Allow CHARS in single-token strings, keep the result

-i,--ignore=CHARS
    Allow CHARS in single-token strings, toss the result

-u,--strip-underscore
    Strip a leading underscore from single-token strings

  C++ language:

-k,--keep=CHARS
    Allow CHARS in single-token strings, keep the result

-i,--ignore=CHARS
    Allow CHARS in single-token strings, toss the result

-u,--strip-underscore
    Strip a leading underscore from single-token strings

  Java language:

-k,--keep=CHARS
    Allow CHARS in single-token strings, keep the result

-i,--ignore=CHARS
    Allow CHARS in single-token strings, toss the result

-u,--strip-underscore
    Strip a leading underscore from single-token strings

  Assembly language:

-c,--comment=CHARS
    Any of CHARS starts a comment until end-of-line

-k,--keep=CHARS
    Allow CHARS in tokens, and keep the result

-i,--ignore=CHARS
    Allow CHARS in tokens, and toss the result

-u,--strip-underscore
    Strip a leading underscore from tokens

-n,--no-cpp
    Don't handle C pre-processor directives

  Text language:

-i,--include=CHAR-CLASS
    Treat characters of CHAR-CLASS as token constituents

-x,--exclude=CHAR-CLASS
    Treat characters of CHAR-CLASS as token delimiters

  Perl language:

-i,--include=CHAR-CLASS
    Treat characters of CHAR-CLASS as token constituents

-x,--exclude=CHAR-CLASS
    Treat characters of CHAR-CLASS as token delimiters

-d,--dtags
    Include documentation tags

Lisp language:

REPORTING BUGS
Report bugs to bug-idutils@gnu.org

SEE ALSO
The full documentation for xtokid is maintained as a Texinfo manual. If the info and xtokid programs are properly installed at your site, the command

    info xtokid

should give you access to the complete manual.