# This document is in Kwid format.  To read this, use a Kwid formatter,
# like "perl6doc perlkwid".

= NAME

perlkwidspec - Normative definition of Perl 6 KWID inline documentation

= DESCRIPTION

This document is a semi-formal definition of KWID.  It is not intended
to be a verbose, blow by blow explanation of everything from how
encodings work to how to tie your shoelaces; some understanding of How
Stuff Works&trade; in general will be assumed from place to place, in
order to reduce redundant documentation.

You will notice that KWID is defined in its own markup language.
Aren't re-entrant specifications great?

= PURPOSE

The design intent of KWID is to:

* provide an /in-situ/ documentation system for Perl 6 (or any
  language, for that matter)

* not /clutter up code/ with needless verbose markup markers

* provide all the /benefits/ of having a verbose markup system

* allow for convergance of documentation, code, and other interesting
  applications of inline documentation clever people have devised,
  such as testing, via a well-designed plug-in system

* allow for alternate styles of text-based formatting as preferred by
  different people, via plugins.

= THE KWID OBJECT MODEL (KWOM)

The KWID Object Model is a /conceptual/ representation of a parsed and
valid KWID document.  A KWID bytecode tree can be thought of as the
closest representation of a KWOM, but is not a KWOM.

It operates in precisely the same way as an XML DOM tree, for
instance.  Nodes are nodes, text chunks are nodes, text nodes cannot
contain nodes, but all other nodes can, text nodes cannot have
attributes (meta-data), but other nodes can, etc etc.  But with the
cruft canned as deemed appropriate.  The details of how this is
accessed or traversed is a detail of the tree module which the current
KWOM implementation uses.  No doubt the KWOM implementation that ships
with pugs will become a /de facto/ standard for this.  However, if you
prefer, you can think of everything in this specifications in terms of
an XML DOM.  Your perception of reality is, after all, entirely
subjective :->.

This may sound strange, but this implies that the KWID markup format
itself is not to be considered normative.  KWID is merely a compact
form of specifying the normative form, which is KWOM.

It is therefore fully possible that at any given time there will be
KWOM documents that cannot themselves be represented in default KWID
format.  These documents may be specified in source via the normative
XML form, until the KWID format is updated to allow such documents to
be specified natively.

So, here is the normal order of flow;

  +-----------+          +-----------+         +----------+
  | Perl6     |------=>  | text KWID |         | KWOM XML |
  | = xxxx    |     /    | form with |         +----------+
  | =cut      |    |     | "anchors" |               //
  +-----------+    /     +-----------+              //
                  |           ||                   //
  +-----------+   /      < KWID Parser >   < KWOM XML Parser >
  | C (etc)   |  |            ||                 //
  | /*        | /             \/                //
  |  = xxxx   |/      +-----------------+      //
  |  =cut     |       |  NORMATIVE KWOM |<=====/
  | */        |       | bytecode format | 
  +-----------+       +-----------------+ 
                              ||
		      < KWOM processing >
                              ||
                              \/
                     ++-----------------++
                     ||   OUTPUT KWOM   ||
                     || bytecode format ||
		     || (with binaries) ||
                     ++-----------------++

As this document model is based on the DOM in concept, *all* KWOM
structures may be converted to *normative XML form*, before they are
processed.  This is described and used for examples later, and will
frequently be referred to as *Normative KWOM XML*.

It is possible to write KWID in source directly in normative XML, via
the `=kwom` starting marker.  However, it should be noted that the
general consensus among programmers is that XML looks horrid.  And
smells.  Note that the connection from KWOM XML to KWOM bytecode
format is the only arrow drawn backwards in the above diagram.

The *OUTPUT KWOM* form is the result of processing of plug-ins, and
will not contain /non-core/ KWOM nodes.  Plug-ins will generally
insert these non-core nodes into the KWOM bytecode tree.

The *OUTPUT KWOM* form is also a KWOM tree.

== Why we bother with all this

SGML is a tried and tested markup form.  This leaves a large amount of
solutions, in various problem spaces subject to leverage.

To provide a concrete example, XPath could provide a lot of
interesting ideas about how to specify links within a document; they
are vastly superior to both POD-style `L<>` links, as well as HTML
anchors.

The other reasons as to why we're converting to a normative form that
has sane and /simple/ characteristics are a little more difficult to
explain `;-)`.

Shortly stated, it is to ensure that all alternative output forms from
`kwid --to_xxx` forms will be represented in a consistent manner.

Also, it is to try to convince people that just because a markup form
is not particularly "regular", it is not completely Evil.  Just as
Perl releases the bounds of being "context free" from programming,
KWID releases the bounds of being "well formed" from documenting.

== Conversion from KWOM to KWID

It is only guaranteed that KWOM documents that were created by the
KWID parser may be converted back to original text form.

This text form may or may not have minor differences "irrelevant to
the spirit of KWID" from the input.  It follows from this that where
multiple equivalent markup forms are allowed in the source, it can be
expected that /metadata/ (represented as attributes in the normative
XML) will indicate which form was used, so that the original input can
be be closely enough represented to satisfy everyone but `md5sum`.

For example, free-form paragraphs will probably be re-flowed, and even
verbatim sections might lose information about which form of
whitespace was used in the input form.  This might, of course, be
subject to feedback from relevant whitespace/wrap width lynch mobs.
ASCII tab characters will always be considered a contentious area, and
hence be left well alone.

== Why the normative XML will differ from `kwidi --to_xml` output

The normative XML is a direct representation of the KWID bytecode
tree.  `kwid --to_xml`, on the other hand, returns an XML document based on
the output KWOM tree, which is suitable for further processing by
native XML tools.

To elucidate this difference with examples;

* a binary image plug-in for very small images will probably show the
  raw binary data (subject to valid XML representation), or the raw
  input in the normative form, but in the output XML form will
  probably be delivered as a seperate file and a node with a URL
  pointing to that file.

* any include directive, to include "stock" sections of documentation,
  such as standard command line options for scripts, etc, would
  resemble the input form in the normative XML, but when processed by
  `kwid --to_xml`, will display the actual included content.  Of course,
  these might be preceded by XML comments or suchlike to indicate
  where they came from.

* output forms will probably be missing line and file number anchors
  essential for being able to stitch code back together from internal
  parse trees.  This may or may not refer to the output KWOM tree,
  depending on how straightforward the implementation end ups being
  ;-)

So, sadly - well, actually, gladly - you won't just be able to dump
normative XML straight to a file, knock up a CSS for it and then
publish it online.  That is what `kwid --to_xml` is for.

= THE 'OUTPUT' KWOM FORM

The internal KWOM bytecode form is not entirely useful on its own in
ensuring that alternative output representations are represented
consistently.

For example, if you encountered an include directive in the KWOM
document, how would you know that you need to load in a seperate file
and then process that?

So, this must all be transparent to the people using the KWOM tree.
For most flexibility, this might be implemented as;

* Allow a pre-processed document to be extracted, that represents the
  fully processed form of the document, using only `native` KWOM node
  types, with associated binary objects.

* Allow the raw tree to be traversed as if it were the expanded form,
  without actually generating an intermediate document, lazily
  performing pre-processing as necessary.

This is probably going to be implemented in terms of a `.processed`
property of the root KWOM tree, which returns another KWOM tree which
will return the results of the preprocessing.  Either implementation
will therefore be equivalent from a usage point of view.

`kwid --to_xml` will probably have an option for dumping the expanded KWOM
form; after all, it will (as all other KWID output generators) treat
this output form as, well, intended for output and therefore the base
for its output :-).  This might even be the default behaviour of
`kwid --to_xml`.

= IMPLICATIONS FOR KWID PLUG-INS

Plug-ins must supply their additions to the KWOM tree, they cannot
merely put a placeholder and a Perl or C object pointer and leave it
at that.  At least, they should be serialisable to KWOM bytecode.

Plug-ins might perform radical alterations to parts of the KWOM tree
which follow; however, in order to ensure that lazy evaluation can
proceed, these cannot affect nodes that appear "earlier" in the KWOM.
It still stuffs up lazy evaluation and performance, to some degree -
for it means that *another* intermediate tree must be generated for
the plugin to #%&@# with.  For this reason, that action of plugins
will probably be strongly discouraged or even banned.

If an object is supplied inline, its data *must* be kept in the
normative KWOM tree.  If the actual object is not supplied inline, but
instead supplied via an externally included object, it *must not* be
kept in the normative KWOM tree.

It should be noted that a plug-in might simply represent itself in
bytecode as an opaque block of text; this is fine, but will make
referring to specific parts of it, as defined later under KWINK,
difficult or impossible.

This includes KWID include directives which are logically treated as
being inserted at that point in the document stream; 

This also means that plugin inclusion cannot return "unbalanced"
fragments.

= KWOM NODE TYPES

T.B.C, based on the different things listed in perlkwid.kwid.

Note that there will be "One True&trade;" node types for tables, even
though people can't agree on the best way of representing those in
text, and hence it won't be possible to specify them without using one
of the various table plug-ins people have dreamt up.

= KWID MARKUP FORM

At this stage, perlkwid.kwid is the best reference.  It may eventually
be defined in this or another document.  But when it is, it will be
defined in terms of what KWOM nodes its markup represents.

= BIG ISSUES WITH THIS DOCUMENT

From Sam Vilain, author of the first version:

The biggest open question in my mind is whether or not the "Output"
KWOM form will be enough to satisfy all markup requirements for KWOM
forever.  I am familiar with many uses of Pod for such things, which
I believe the above caters for nicely, but I have not personally
scoured CPAN for other uses people have found.

On the one hand, allowing the expanded KWOM form to have "attachments"
should cater for this pretty much without limitation so there should
be ample room for people to make out-of-band hacks for specific output
formats.

On the other hand, documentation should be simple, so anything that
doesn't fit the DOM should be politely invited to leave by means of
fornication.  After all, there are always source filters (or their p6
equivalent) if people want to define their own markup system.

It would be nice to be able to say that the KWOM tree could be built
without having plug-in implementations on hand, but realistically I
doubt this will ever be possible, at any level!

Another question is, would intra-document links link to the original,
or processed KWOM documents?  I would think the former...
