# This document is in Kwid format.  To read this, use a Kwid formatter,
# like "perl6doc perlkwid".

=dialect kwid

= NAME

perlpodspec - Perl 6 POD and POD dialects normative specification

= DESCRIPTION

This document is a semi-formal definition of POD.  It is not intended
to be a verbose, blow by blow explanation of everything from how
encodings work to how to tie your shoelaces; some understanding of How
Stuff Works&trade; in general will be assumed from place to place, in
order to reduce redundant documentation.

You will notice that POD is defined in its own markup language.
Aren't re-entrant specifications great?  In fact, it's worse - it's
defined in a /dialect/ of it's own language!

That dialect is called Kwid, which is by hereby decreed to be the best
POD dialect there is.

This document contains lots of design questions and explanations that
may not belong in this document when the standard is settled.

= PURPOSE

The design intent of POD is to:

* provide an /in-situ/ documentation system for Perl 6 (or any
  language, for that matter)

* not /clutter up code/ with excessive markup

* allow for alternate styles of text-based formatting as preferred by
  different people, via POD /dialects/.  Multiple dialects should be
  able to be used at the same time.

* provide all the /benefits/ of having a verbose markup system like
  XML, for instance by making the documents effortlessly parsable by
  other systems, even in the face of arbitrary dialects

* allow for convergance of documentation, code, and other interesting
  applications of inline documentation clever people have devised,
  such as testing, via a well-designed parsing system.

= THE POD ABSTRACT SYNTAX TREE (AST)

The POD Abstract Syntax tree is a representation of a parsed and valid
POD document, which may or may not include nodes that are of POD
dialects.  These entities are referred to in the rest of this document
as /POD ASTs/.

They are very close in behaviour and manner as an XML DOM tree, for
instance.  That is; Nodes are nodes, text chunks are nodes, text nodes
cannot contain nodes, but all other nodes can, text nodes cannot have
attributes (meta-data), but other nodes can, etc etc.

The details of how this is accessed or traversed is a detail of the
tree module which the current POD AST implementation uses.

No doubt the POD AST implementation that ships with pugs will become a
/de facto/ standard for this.  However, if you prefer, you can think
of everything in this specifications in terms of an XML DOM.  Your
perception of reality is, after all, entirely subjective :->.

It is therefore fully possible that at any given time there will be
POD AST that cannot themselves be represented in the format of a
particular dialect.  These documents may be specified in source via
the normative XML form (described below), which is always processed in
exactly the same way, despite the presence of an active dialect.

So, here is the normal order of flow;

  +-----------+          +-----------+         +---------+
  | Perl6     |------=>  | text POD  |         | POD XML |
  | =xxxx     |     /    | form with |         +---------+
  | =cut      |    |     | "anchors" |               //
  +-----------+    /     +-----------+              //
                  |           ||                   //
  +-----------+   /      < POD Parser >    < POD XML Loader >
  | C (etc)   |  |            ||                 //
  | /*        | /             \/                //
  |  =xxxx    |/      +-----------------+      //
  |  =cut     |       |  NORMATIVE POD  |<=====/
  | */        |       | AST (or stream) | 
  +-----------+       +-----------------+ 
                              ||
		      < pre-processing >
                              ||
                              \/
                     ++-----------------++
                     ||   OUTPUT POD    ||
                     || AST (or stream) ||
		     || (with binaries) ||
                     ++-----------------++

As this document model is based on the DOM in concept, *all* POD
documents may be converted to *normative XML form*.  This form is
described and used for examples later, and will frequently be referred
to as *Normative POD XML*.

It is possible to write POD or even Kwid in source directly in
normative XML, via the `=xml` starting marker.  However, it should be
noted that the general consensus among programmers is that XML looks
horrid.  And smells.  Note that the connection from POD XML to
Normative POD is the only arrow drawn backwards in the above diagram.

The *NORMATIVE POD* form will have already been parsed for validity by
available plug-ins, and its paragraphs broken into POD AST nodes where
they contain formatting, etc.  It is necessary to perform this right
at the beginning, because the Parser ensures that it is emitting a
"balanced" set of nodes by emitting dummy closing nodes when none are
found in the original source.

The *OUTPUT POD* form is the result of processing of plug-ins, and
will not contain /non-core/ POD nodes, such as Kwid.

The *OUTPUT POD* form is also a POD AST.

== Why we bother with all this

SGML is a tried and tested markup form.  This leaves a large amount of
solutions, in various problem spaces subject to leverage.  However,
the representation itself is not important; the nice object model is
more so.

To provide a concrete example, XPath could provide a lot of
interesting ideas about how to specify links within a document; they
are vastly superior to both POD-style `L<>` links, as well as HTML
anchors.

The other reasons as to why we're converting to a normative form that
has sane and /simple/ characteristics are a little more difficult to
explain `;-)`.

Shortly stated, it is to ensure that all alternative output forms from
`kwid --to_xxx` forms will be represented in a consistent manner.

Also, it is to try to convince people that just because a markup form
is not particularly "regular", it is not completely Evil.  Just as
Perl releases the shackles of being "context free" from programming,
POD dialects, like Kwid, releases the shackles of being "easily
parsable" from documenting.

== Conversion from POD ASTs to source form

It is only guaranteed that POD ASTs that were created by the POD
parser without /pre-processing/ may be converted back to original text
form.

This text form may or may not have minor differences "irrelevant to
the spirit of POD" from the input.  It follows from this that where
multiple equivalent markup forms are allowed in the source, it can be
expected that /metadata/ (represented as attributes in the normative
XML) will indicate which form was used, so that the original input can
be be closely enough represented to satisfy everyone but `md5sum`.

For example, free-form paragraphs will probably be re-flowed, and even
verbatim sections might lose information about which form of
whitespace was used in the input form.  This might, of course, be
subject to feedback from relevant whitespace/wrap width lynch mobs.
ASCII tab characters will always be considered a contentious area, and
hence be left well alone.

== Why the normative XML will differ from `p6d --to_xml` output

The normative XML is a direct representation of the normative POD AST
tree.  `p6d --to_xml`, on the other hand, returns an XML document
based on the output POD AST, which is suitable for further processing
by native XML tools.

To elucidate this difference with examples;

* a binary image plug-in for very small images will probably show the
  raw binary data (subject to valid XML representation), or the raw
  input in the normative form, but in the output XML form will
  probably be delivered as a seperate file and a node with a URL
  pointing to that file.

* any include directive, to include "stock" sections of documentation,
  such as standard command line options for scripts, etc, would
  resemble the input form in the normative XML, but when processed by
  `p6d --to_xml`, will display the actual included content.  Of course,
  these might be preceded by XML comments or suchlike to indicate
  where they came from.

* output forms will probably be missing line and file number anchors
  essential for being able to stitch code back together from internal
  parse trees.  This may or may not refer to the output POD AST,
  depending on how straightforward the implementation end ups being
  ;-)

So, sadly - well, actually, gladly - you won't just be able to dump
normative XML straight to a file, knock up a CSS for it and then
publish it online.  That is what `p6d --to_xml` is for.

= POD STREAM EVENT PROCESSING

The internal POD AST form is not entirely useful on its own in
ensuring that alternative output representations are represented
consistently.

For example, if you encountered an include directive in the POD AST,
how would you know that you need to load in a seperate file and then
process that?

So, this must all be transparent to the people using the POD AST.  To
ensure this, a system of event pre-processing will be used.

The `Pod::Parser` (or any class with the /Role/ of `Pod::Emitter`,
like `Pod::AST`) will emit POD paragraphs (strictly, POD /events/) to
a class that supports the `Pod::Loader` Role.

The events coming out of `Pod::Parser` will /already/ be parsed into
POD AST events.  This happened because `Pod::Parser` took the raw
paragraphs and used its rules and dialect state to find an appropriate
Dialect module to convert it to a series of POD AST events.

`p6d --to_xml` will probably have an option for dumping the expanded
POD form; after all, it will (as all other POD output generators)
treat this output form as, well, intended for output and therefore the
base for its output :-).  This might even be the default behaviour of
`p6d --to_xml`.

== THE NORMATIVE POD AST

Dialects must parse their paragraphs into proper POD AST nodes, which
are completely serialisable to XML; they cannot merely put a
placeholder and a Perl or C object pointer and leave it at that.  They
should be structural break-down, and nothing else.  Nodes which were
handled by dialects, and hence have further pre-processing to be
performed on them, are special types of nodes.

A Normative POD AST is still completely serialisable back to its
original source form.

If an object is supplied inline, its data *must* be kept in the
normative POD AST.  If the actual object is not supplied inline, but
instead supplied via an externally included object, it *must not* be
kept in the normative POD AST.

It should be noted that a dialect might simply represent itself in the
POD AST as an opaque block of text; this is fine, but will make
referring to specific parts of it difficult or impossible.

This includes POD include directives which are logically treated as
being inserted at that point in the document stream.

== THE OUTPUT POD AST

Plug-ins which perform pre-processing might perform radical
alterations to the stream of POD events which follow it (or, if you
prefer, the regions of the POD tree which follow it).  The
pre-processing system will allow plug-ins to "catch" arbitrary parsing
events that follow it, and either pass them through to the next
pre-processor, or do something with them itself.

The output from this preprocessing will be a series of POD AST nodes
that does not include any "special" nodes.  If the Dialect
pre-processing for a node returns a "special" node, pre-processing
will continue, subject to a maximum recursion limit.

This also means that plugin preprocessing or loading *can* return
"unbalanced" events, but the pre-processor or loading will be
responsible for ensuring that all loaders receive balanced events on
input.

Preprocessing can also emit binary objects as attachments, which might
get delivered to a file or perhaps will be rendered straight away.
These will travel through the output stream.

= POD NODE TYPES

Basically, everything in the POD AST, + tables.  TBC.

= KWID MARKUP FORM

At this stage, perlkwid.kwid is the best reference.  It may eventually
be defined in this or another document.  But when it is, it will be
defined in terms of what KWOM nodes its markup represents.

= BIG ISSUES WITH THIS DOCUMENT

From Sam Vilain, author of the first version:

The biggest open question in my mind is whether or not the "Output"
POD AST form will be enough to satisfy all markup requirements for POD
preprocessing forever.  I am familiar with many uses of Pod for such
things, which I believe the above caters for nicely, but I have not
personally scoured CPAN for other uses people have found.

On the one hand, allowing the expanded POD form to have "attachments"
should cater for this pretty much without limitation so there should
be ample room for people to make out-of-band hacks for specific output
formats.

It would be nice to be able to say that the POD AST tree could be
built without having plug-in implementations on hand, but
realistically I doubt this will ever be possible.

Another question is, would intra-document links link to the original,
or processed POD ASTs?  I would think the former...
