TODO list for Pugs-Compiler-Perl6

- parse ext/Test/lib/Test.pm
  - &?CALLER::ROUTINE
  - s:P5:g/foo/bar/;
  - :$flag in capture
  - runtime support for caller().package and &?ROUTTINE.name
  - $TEST::foo in Test.pm needs to be mangled as well like the namespace.

  - named arg binding with Data::Bind - (done on the pX/clkao branch)

- block parsing:
  t/ref.t: if $string.ref eq Str { say "ok 1 # TODO" }
  <audreyt> for map {},@x { .say }

pre-release - jun/2006

- cache compiled YAPP and Rule code, use Module::Compile
- write POD
    - v6.pm 
    - Pugs::Compiler::Perl6->compile
    - Pugs::Grammar::Perl6->parse
- add an option to emit ast

Possible Backend Modules

- Moose, Moose::Meta
- Class::Multimethods::Pure

(older TODO - needs a revision)

Priorities 

- Things that are not giving error messages:
    (1 if 1)           # 'if' is an op - I think this is now legal 
    sub sub xxx ...    # 'multi' and 'sub' are parsed the same way

- failures:
    - it is trying to parse the examples in the pod
- workarounds:
    - listop if ...;  -- implemented as a special case
    - listop;    -- say;
    - $obj.op()  -- when 'op' is a declared operator (not plain bareword)
    - prefix/infix:<&> vs. &code
    - <...> and %x<...> should be implemented as circumfix and postcircumfix op
    - /rule/ is a term
  
- merge Pod.pm into Perl6.pm

- check that these special cases are covered:
    moose=>1
    moose:<elk>
    moose:{antler()}
    sub 'ternary:?? !!'

- transform hash collisions into alternations
    example: short name of prefix:<&> (Prefix.pm) vs. sigil in &name (Var.pm)

- missing syntax
    elsif

TODO

- implement statements using top-down parser (rules)
- implement ordered testing of categories <%a|%b|%c> depending on the parser state
- expression terminator (maybe the tokenizer is the best place?)

- parse the op table from
  http://svn.perl.org/parrot/trunk/languages/perl6/lib/grammar_optok.pge
- use the grammar from
  http://svn.perl.org/parrot/trunk/languages/perl6/lib/grammar_rules.pge

- macro
- dynamic grammar

- unify syntax in Operator.pm:
    unless/for
    operators with the same precedence, fixity, and associativity
    subroutines
    sub/multi
    BEGIN/END

- Implement compile-time dynamic 'add_token' (lexical at run-time)

- Implement double-quoted-string split on variables (interpolation)
- Implement /rule/ and '/'
    - see expect... in Expression.pm

<fglock> TimToady: is the lexer the right place to make the '<'/'<a>' distinction? (it is not user-modifyable)

<TimToady> fglock: that depends on what you count as part of the lexer.  
    The bottom-up parser knows when it's looking for a <%infix> vs <%prefix> vs <%postfix>, so only those tokens are active that would be valid at that spot.
<TimToady> (I oversimplify the hash stuff slightly there.)
<TimToady> It's more like:
<TimToady> <%infix> vs <%prefix|%term|%circumfix> vs <%postfix|%postcircumfix>
<TimToady> assuming we adopt the new <%a|%b|%c> notation to combine 
    longest-token processing of multiple hashes.
<TimToady> fglock: for speed one could cache all the hash keys for all the hashes in a trie or some similar structure.  Just have to be careful that longest key wins regardless of hash, and in case of tie first hash wins.
<TimToady> 'course you have to recalculate if any of the hashes is modified...

<TimToady> can probably treat alphanumeric sub names specially so that you don't have to recalculate on every sub declaration.
<TimToady> if you assume that no "foo" prefix operator or term can match if the next char is alphanumeric.
<TimToady> maybe just run the prescanned identifier down a different trie than the non-alpha ops.
<TimToady> actually, if you know the length then the ident one doesn't need a tree.  Just a hash would work.
<TimToady> since you know its length already.

<fglock> TimToady: what if both postcircumfix and infix are expected? then the op is chosen based on if there is whitespace or not?
<fglock> like in %ENV<x> vs. %ENV <...
<TimToady> <%postcircumfix|%infix> is what you look for before whitespace, and <%infix> after.
<TimToady> that's why we completely outlawed whitespace before postfix.
<TimToady> hmm, that doesn't quite work.
<TimToady> I think at postfix location you actually look for <%postfix|%postcircumfix>|<%infix> becuase
<TimToady> you don't want the %infix participating in longest token there.
<TimToady> $x<=2 is an error, but $x <= 2 is okay.
<TimToady> or looking at it in terms of whitespace, if you don't get any match on a postfix, then you can pretend there was whitespace even if there wasn't, and try %infix.
<fglock> I think I'll need to do some tests ... - how about /rule/ vs. division? is it just that rule is a term and division is an op?
<TimToady> yeah, that's just simple term vs op expectation.
<TimToady> just as in P5.
<TimToady> It's really only the postfix category that's new to P6

<fglock> TimToady: is specifying an 'end token' parameter the way to go in interfacing the rule-based statement parser to the bottom-up parser?
<TimToady> "an" end token is probably limiting it unnecessarily.  A set of end tokens is more like it.
<TimToady> But a match can match a set of tokens, so passing in an end match is probably sufficient.
<TimToady> The real problem with lvalue substr is that Perl 5 fakes it without a real COW engine underneath.
<fglock> are the things inside brackets parsed using subrules? like in (@a;@b); - so that the 'end' is not matched in the wrong place
<TimToady> As a basic rule of thumb, if you have brackets, you probably want a subrule
<TimToady> though it's possible to do with op prec if you fiddle with infinities in the relative precedence inside and outside.
<TimToady> but it's difficult to detect the (@a;@b] error without knowing the "right" terminator.
<fglock> is sub 'circumfix:( ]' a valid spec?
<TimToady> It's sort of like the difference between which things you need to use recursive regex on vs which things you can match with a flat regex (in P5 terms).
<TimToady> syntactically, yes, that's valid.  You're nuts if you do it though.
<TimToady> And you'll likely get a warning anyway if you define two circumfixes with the same left side.
<TimToady> s/two/a second/
<TimToady> but there's nothing says a circumfix can't be <fred barney>

- Implement [op]@list
- Implement expressions inside names - like:
    prefix:{'+'}

- Make the tokenizer match eagerly (faster)

- Implement the "magic hash" dispatcher

    TimToady on #perl6 -
    xxx:<+> has to be considered just a funny looking name. 
    It's the grammar's responsibility (somehow) to pull in any existing xxx and 
    newly created xxx and combine them into any rules or %hash that references them.
    supposing a grammatical category shows up in %xxx, then we need two ways to deal 
    with it.
    first, if we want one category to hide another, you can get away with a mixin 
    style of
        rule { %xxx | %yyy | %zzz }
    but there are some syntactic categories that have to be magically combined like
    compile-time roles:
        rule { %xxx_or_yyy_or_zzz }
    that is, the longest-token rule is applies in parallel across all the categories
    simultaneously.
    that's why the magic hash was invented (or more accurately, is scheduled to 
    be invented :)

-   the tokenizer should get tokens lazily ?

-   is 'space-{' is found, is sent to the opp - if the opp is expecting an operator,
    it means end-of-expression

    TimToady in #perl6 - space + block is a top-level block only where an operator 
    is expected, and you're not in brackets.
    where a term is expected, it's just a closure argument. (or a hash composer)

- Specify/generate AST

P::C::R BUGS

- A Match doesn't stringify if there is a capture

- Fix %hash capture - it currently works like this:

  sub capture {
    # print Dumper ${$_[0]}->{match}[0]{match}[1]{capture}; 
    return ${$_[0]}->{match}[0]{match}[1]{capture};
  }

- needs character classes

- needs <after...> (update: implemented in :ratchet)

- cleanup Pugs::AST::Expression (no longer used?)

- Token/Rule/Regex have problems calling each other
