This file contains assorted design notes from #perl6 or docs - fglock

from PGE docs: Available syntactic categories include "infix:", 
"prefix:", "postfix:", "term:", "circumfix:", and "postcircumfix:"

 --> and: statement_control ?

random notes...

<fglock> audreyt: what is the relationship between AST and Match? (I'm compiling 
the match captures)
<audreyt> fglock: no relationship whatsoever :)
<audreyt> fglock: the Match object may carry an captured object in $/<>
<audreyt> aka $()
<audreyt> and if you are writing a Perl 6 parser, then that capture object may be 
an AST object
<audreyt> you can set the capture object by
<audreyt> rule { $() := ... }
<audreyt> or
<audreyt> rule { ... { return $capture_object } }
<audreyt> or
<audreyt> rule { ... { $() := ... } }
<audreyt> if the capture object is not set explicitly
<audreyt> then it's set to the entire match as matched by the rule.
<spinclad> so in q:code:{ say $x; {{{$a}}} } the $x is literal but the $a is 
unquoted (interpolated)? therefore the {{{ }}}'s?
<audreyt> +$/ and ~$/ resolves to +$() and ~$() respectively.

----

re: '&'

22:16 < putter> [<mail_header>&.*?^^From: (\N+).*] $<body>:=(.*)
22:20 < putter> PerlJam: what's an increased specificity example?  both 
of mine
                were assert-spec&unpack pairs.
22:22 < fglock> putter: .* would make it fail, isn't it?
22:23 < putter> you mean the .* after the (\N+) ?
22:23 < fglock> yes
22:23 < PerlJam> putter: yes, mine are similar, just that the "unpack" 
part
                 isn't necessary:    [<mail_header>&^^<'From:'>] &&
                 do_something_only_with_from_lines;
22:23 < putter> I don't believe so.  I would expect the & to force the * 
to
                backtrack.
22:24 < PerlJam> putter: btw, beware the cut-colon!  :-)  (unless I'm 
mistaken
                 that you meant for the : to be matched)

-----

re: statement_control

22:50 < putter> one uses  multi statement_control:<if> (...){...}  multi
                statement_control:<while> (...){...} etc to fill it in.
22:51 < fglock> putter: you mean, statement_control is represented by an 
array?
                (or namespace thing)
22:52 -!- pdcawley [n=pdcawley@adsl-217.146.110.1.merula.net] has joined 
#perl6
22:52 < putter> fglock: yeah, but there are some issues... like how do  
(pause)
22:52 -!- clkao [n=clkao@220-132-58-30.HINET-IP.hinet.net] has quit 
[Read
          error: 104 (Connection reset by peer)]
22:52 < PerlJam> must not have been enough svk talk here ;-)
22:52 < fglock> putter: please not that it is an array of rule - it is 
very
                flexible
22:53 < putter> statement_control is a grammatical category.  it defines 
one of
                the <statement> subrules.
22:53 < putter> (my yeah, was directed at <@subrule>, not represented by 
an
                array ;)
22:55 -!- r0n|mess [n=r0nny@p54B893E3.dip0.t-ipconnect.de] has quit 
[Connection
          timed out]
22:57 < putter> fglock: yes but.  when writing a grammar, in a first 
match wins
                engine (like | is), you carefully craft the order of the
                subrule list.  when subrules get added by 
statement_control
                defs, someone other than the human has to do the 
crafting.
                either the statement_control infrastructure assures the 
@array
                has a nice order, or <statement> can't use <@array>.
22:58 < fglock> putter: you can opt to use longest-match instead of
                ordered-match
22:58 < putter> yes
22:58 < fglock> that would be <%statement_control>
22:59 < putter> and the real parser can play games like trying to 
massage the
                @array into a trie, so it doesnt have to repeatedly 
reparse the
                same stuff the same way.
22:59 < fglock> putter: it is cached

23:02 < putter> re statement_control,
                http://dev.perl.org/perl6/doc/design/syn/S04.html has a 
little
                bit in Statement parsing.

----

re: Smart::Comments
re: defining 'if' with macros

23:08 < putter> re hash, "An interpolated hash matches the longest 
possible key
                of the hash as a literal, or fails if no key matches.", 
which
                doesnt help you distinguish between  /if <expr> <block> 
[else
                <block>]?/ and /if <expr> <block> [else <block>]?
                [wrap_around_both_branches <mumble>]?/.   comparing "if" 
and
                "if" isnt going to help.
23:09 < pmurias> Does any one think that useing Smart::Comments in
                 iterator_engine.pl would be a good idea?
23:09 < fglock> pmurias: I like Smart::Comments, but I'd like to keep it 
simple
                (that is, no unnecessary dependencies)
23:09 < putter> (there shouldn't have been a ? on the <mumble> clause)
23:10 < pmurias> It would be a dependency only for debugging :)
23:10 < pmurias> And casual users don't do that often :)
23:11 < pmurias> at least they shouldn't have to :)
23:11 < fglock> pmurias: I'll check that (you mean, disable 'use
                Smart::Comments' when not in use?)
23:11 < pmurias> the use line should be comment out by default
23:12 < fglock> putter: re /if.../ - I don't understand the mumble part, 
what
                would it be?
23:12 < pmurias> and if the debuging messages are needed you just delete 
the #
23:12 < putter> re if macro, well, you need to add a regex to 
<statement> so
                you can parse it.  and we can currently hang regexs off 
of,
                well, rule, and macros     macro statement_control:<if> 
(...)
                is parsed(/heres the regex/) {...}
23:12 < pmurias> i'll commit it tommorow if youd don't mind
23:12 < fglock> pmurias: sounds good - I'll check the pod again


23:36 < fglock> it will look like: %statement_control:<if> = rule { ... 
}

-------

re: bootstrapping vs. correctness

<TimToady> personally, I'd aim for the bootstrap, and if we have to 
write
<TimToady> "funny" code to use it, that's okayish.
<fglock> putter: :)
<fglock> TimToady: ok - this helps!
<putter> fglock: what are some possible milestones one could imagine 
reaching for?
<fglock> for example: I could implement 'if' very easily in perl5, but 
implementing with a Perl 6 macro is more correct-ish
<fglock> putter: the compiler is not OO, so we don't need that in the 
bootstrap
<fglock> so implementing closures, hash, array should be ok
<fglock> operator precedence isn't important right now
* putter 's next question will be "what is the _simplest_ approach you 
can imagine?"
<fglock> "_simplest_" - keep writing nodes in perl5; "right" - start 
writing Perl 6 from this point
<fglock> however, nodes can be migrated to Perl 6 later, now that we 
know how to do it
* putter is a great fan of simple
<fglock> maybe next step: rewrite iterator_engine.pl in simple-Perl6, 
and struggle to compile it to Perl5
<putter> hmm, another thing one might do is step back, and try to come 
up with a list of _other_ things you might be working on/towards.  
because it's not only is this useful in itself, but is it the most 
useful thing you can think to do at the moment


-----------

re: pads

<fglock> is there some CPAN module that implements a hash that behaves 
something like a pad? we could use it for implementing lexical subs for 
example
<TimToady> Yes, especially if scoped lexically via %^H modifying pragms.
<fglock> pmurias: each node is actually a state machine
<gaal> what ever happened to MJD's pragma patch?
<fglock> s/each node/each node in a rule/
*** Daveman left #perl6 ["Leaving"]
<TimToady> gaal: dunno--I only ever heard about it second-hand--never 
looked at it.
<pmurias> fglock:I sort of understand now :)
<gaal> it's really elegant. (in taking advantage of the 
parser<->evaluator chumminess of p5)
<webmind> .win 3
<webmind> oops
* gaal wanders off to entertain a guest
<webmind> hf
<putter> fglock: 
http://search.cpan.org/~mneylon/Tie-Hash-Stack-0.09/Stack.pm ?

------

re: modifiers

<PerlJam> fglock: rule xxx { :perl5 .* }
<PerlJam> fglock: in yours, .* would be a parameter to the :perl5 
attribute.
*** FurnaceBoy [n=qu1j0t3@67.68.33.193] has quit [brown.freenode.net 
irc.freenode.net]
*** chris2 [n=chris@p549D1C58.dip0.t-ipconnect.de] has quit 
[brown.freenode.net irc.freenode.net]
*** meppl [i=mephisto@meppl.net] has quit [brown.freenode.net 
irc.freenode.net]
<putter> :perl5 , like :i :mumble, is a modifier.
<putter> block scoped
<putter> [:i(1) This CoDe Is CaSe insensiteive ]


----

re: building AST

<fglock> re OO AST nodes - does it make any sense to use hashes, and 
autobox when necessary? (just because building objects during the 
parsing may be expensive - backtracking throws many nodes away)
*** Nouk [n=Nouk@219-87-211-192.static.tfn.net.tw] has joined #perl6
*** Nouk [n=Nouk@219-87-211-192.static.tfn.net.tw] has quit [Client 
Quit]
<audreyt> fglock: for perf I'll even just use arrays with constant 
lookup keys
<audreyt> fglock: but I'd suggest to _not_ care about performance
<audreyt> just optimize for clarity/effieciency for coding
<audreyt> it may be nice to adopt the haskellish
<audreyt> Val(VInt(3))
<audreyt> i.e. instead of calling ->new, use functions as constructors
<fglock> audreyt: but are resulting nodes objects of functions?
<fglock> audreyt: you mean like:   rule xxx {... { return Val(VInt(3)) } 
}
<fglock> audreyt: I'm ok with that - Val and VInt are "node constructor" 
functions, which we can modify later
<fglock>     rule xxx {... { return Val(VInt( $1 )) } }
<audreyt> yes.
<audreyt> this is because in p6
<audreyt> VInt(3)
<audreyt> is basically
<audreyt> 3 as VInt
<audreyt> and you can install multis in the VInt class to anticipate the 
infix "as" call
*** drbean [n=greg@momotaro.chinmin.edu.tw] has joined #perl6
<fglock> audreyt: and if Val returns a function, we can delay "building" 
the AST by not evaluating it, right?
<audreyt> yes
<audreyt> but I'm not sure how much you gain in p5land from that
<fglock> audreyt: I don't know too - just thinking aloud :)
<audreyt> without core support for thunks, and with a typical compiler 
run visiting all nodes
<audreyt> delaying ast buildup doesn't gain you much.
<fglock> audreyt: you get time by not bulding things that would be 
destroyed by backtracking - I think that's all
<audreyt> yup, so it's CV creation vs HVMG creation
<audreyt> well, with the encapsulation offereed by function constuctors 
we can switch style at all times
<audreyt> but my gut feeling is that plain HVMG is good enough
<fglock> audreyt: VInt(3) - you don't need to specify the argument name? 
(sorry, I don't really know p6)
<fglock> HVMG?
<audreyt> no - it's a coercion
<audreyt> blessed hashref
<audreyt> some help in p5land for this kind of thing:
<audreyt> http://search.cpan.org/dist/Symbol-Opaque/
<audreyt> http://search.cpan.org/dist/Data-Variant/

-------------

re: Parser optimization

<TimToady> fglock: the P6 grammar is designed to not require much 
backtracking at all (unless you want to go back and give better diags), 
so I wouldn't sweat it.
<fglock> TimToady: ok - it's mostly because I haven't implemented ':' 
yet :)
*** avar [n=Huld@dsl-228-236.hive.is] has quit [Connection timed out]
<TimToady> fglock: in fact, if there's *any* place in the grammar that 
requires backtracking for a correct parse, I'd like to know about it...
<fglock> TimToady: I'll maybe add a switch to disable backtracking or 
give a warning - so we'll know :)
*** avar [n=Huld@dsl-228-236.hive.is] has joined #perl6
<TimToady> cool
<TimToady> we even try to avoid alternation in favor of <%tokens>.
<TimToady> and <%tokens> is potentially the merge of several syntactic 
categories, so we don't have to say
<putter> is there a target grammar category?  (in the parsing sense, not 
the p6 sense;)
<TimToady> <%statements_control> | <%unaries> | <%terms>
<fglock> TimToady: is there an existing Grammar file you would recommend 
as a start point?
<TimToady> putter: don't understand what you're asking...
* putter wonders what %unaries is
<putter> LALR(1), etc
<TimToady> s/unaries/prefix/
<TimToady> + circumfix left side
<putter> ah, ok
<TimToady> Actually, circumfix could be considered a subset of terms.
<TimToady> fglock: not that I'm aware of, unless Patrick has one.
<TimToady> The approach we've been taking...
*** marmic 
[n=chatzill@c-93b6e055.1258-1-64736c22.cust.bredbandsbolaget.se] has 
joined #perl6
<TimToady> is to have a 3-layer parser.
<TimToady> (well, where the 3rd layer is really fractal...)
<TimToady> the top layer is top-down, down to expression level.
<TimToady> the middle layer is a bottom-up operator precedence parser to 
avoid
<TimToady> excessive recursion through 22 or so prec levels.
<TimToady> each term can then be its own little top-down parser inside 
the token grabber for the op prec parser.
<TimToady> I also worry about efficiency... :)
<fglock> TimToady: :)
<putter> the token grabber selects which term to use how?
<TimToady> One of the additional reasons for installing the op-prec 
parser in the middle is to allow for added prec levels.
<TimToady> but we don't necessarily need that for a bootstrap.
<TimToady> Unless we actually add all of Perl 6's opcodes that way.
<fglock> TimToady: the idea (so far) is to add all prims using sub or 
macro declarations
<TimToady> <%mush_of_all_syntactic_categories_where_term_is_expected>
<TimToady> which can have prec declarations, so we could add most prec 
levels that way, if the underlying prec architecture can support it.
<putter> terms can have prec decls?
<TimToady> terms by definition have a "squinting" prec that is 
infinitely tight looking out and infinitely loose looking in.  That's 
why circumfixes are so popular to represent visually the "surreal" 
precedence change.

<bsb> the args to macros should always be ASTs, right?
*** justatheory [n=justathe@70.103.133.226] has joined #perl6
<fglock> What is the default precedence when you create a sub? (tighter 
than anything else?)
*** vodka-goo [n=dbaelde@129.104.11.1] has quit ["Leaving"]
<TimToady> bsb: depends on what the "is parsed" rule returns, and the 
signature declares, and whether they match.

<TimToady> by default subs are list operators.

<TimToady> except ($) is still unary op, last I checked.
<TimToady> but that might be worth breaking...
<fglock> TimToady: '$'?
<TimToady> single scalar argument in P5-proto-speak.
<bsb> I had an idea about using unary macros to do haskell's comma-less, 
tight, currying function calls...
<TimToady> could be made to work.
<bsb> is parsed gives the macro's grammar category, right?
<TimToady> I believe at one time we made infix:<X> default to the same 
prec as infix:<+>, but lately I think we just require "is equiv" or some 
such.
<fglock> bsb: macro infix:<xxx> ... - the grammar category comes before
<TimToady> eh?  the grammatical category is part of the macro's name, or 
I'm misunderstanding your question.
<putter> inside a circumfix, are the trailing tokens of surrounding 
circumfixi still tokens?
<bsb> I might just slink away from that question and get to the thing 
that's confused me...
<TimToady> The inner rule has to be parameterized to know what its 
terminators are, if they would be confused with inner tokens.
<bsb> After talking to audreyt, I thought macro's arguments were always 
ASTs, but the macros in Prelude seem to expect normal values
<TimToady> but usually any unbalanced bracket could be assumed to 
terminate.
<TimToady> ASTs are certainly encouraged as the default, much as opaque 
objects are now the default in P6.  But we want to leave the hooks there 
to do something else.
<bsb> That means the Prelude should probably revert to subs instead of 
macros for now
<TimToady> terminator params might actually be one of the places we 
choose to backtrack rather than merging categories:
<TimToady> <%terminators> | <%expect_term>
<TimToady> since the merge would have to be redone on every subrule 
call.
<fglock> TimToady: you mean rules inside 'is parsed' may need to take 
the terminator as a parameter in order to know where to finish macthing?
<bsb> putter: ping
<putter> circumfix:<+ +>    +(3+2)+  ?
<putter> bsb: pong
<TimToady> fglock: yep, if the rule is sufficiently antisocial to want 
conflicting things.
<TimToady> though I see it's actually
<bsb> gaal said you know about the macos in Prelude, can they be 
reverted to subs?
<TimToady> <%terminators> | <%expect_operator> from the +...+ example.
<fglock> TimToady: the problem is there can be any rule inside the 'is 
parsed' - and then all rules would need to check is this parameter is 
set
<fglock> I mean, the macro parameter can be any rule
<bsb> until we have macros params that can be ASTs or normal values
<TimToady> yes, perhaps it's really something environmentally magical 
that all <%foo> just automatically track.
<fglock> TimToady: ok
*** p2502 [n=p2501@84.12.179.136] has quit [Read error: 104 (Connection 
reset by peer)]
<TimToady> Anyway, for well behaved subrules, we can just assume 
trailing brackets for now, as long as we keep the generalization in 
mind.
<fglock> TimToady: I don't see '^' in the example Grammars - is it 
automatic (to match on the start of the string)
<TimToady> Maybe we can semi-generalize that to "stop on syntax error", 
but that bugs me a little, since there are real syntax errors as well as 
fake ones...
<TimToady> subrules are always anchored on the left to where their 
parent rule called them.
<TimToady> But some of them may also be marked :p to start at the 
current pos.
<TimToady> but it's possible this will turn into a token vs rule 
distinction on the declaration itself.
<fglock> TimToady: is 'token' a new keyword?
<TimToady> conjectured.

* putter fails to parse "subrules are always anchored on the left to 
where their parent rule called them. BUT some of them may also be marked 
:p to start at the current pos."  not equivalent sentences?
<TimToady> Interestingly, I see a def for "token" in S06 but nothing 
discussing it...

<fglock> putter: :p is optional (that's not a tongue)
<TimToady> not equivalent when the subrule is called directly rather 
than as a subrule.

<TimToady> if we had a "token" keyword, it might autoanchor on both ends 
when called directly on a string rather than as a subrule, unlike :p 
which is currently specced as just the front anchor.  But it still needs 
some thought...

<putter> is there any conjectured nomenclature for token,definition-of 
vs token,instance-of?  (ie, class names:)
<TimToady> well, if a rule is a Rule, then a token would be a Token, I 
expect.
<TimToady> I've got to head off to $job now.
<putter> thanks for the puzzle answers :)
<TimToady> I don't know if they're answers, but maybe they're at least 
better questions.

---------

re: CPAN

<audreyt> or, hell, there's no Pugs::* anything yet
<audreyt> so let's just use that
<audreyt> Pugs::Grammar, Pugs::Optimizer
<fglock> audreyt: ++
<Juerd> audreyt: And maybe, Pugs::Prelude?
<audreyt> aye
<Juerd> audreyt: Which would be nice on CPAN, I think :)
<audreyt> yeah.
<fglock> audreyt: how about the plug-in thing?
<audreyt> fglock: follow the Module::Pluggable tradition
<audreyt> Pugs::Optimizer::Plugins::ThisPluginName
<nothingmuch> what about freepan?
<nothingmuch> isn't that singular Plugin?
<audreyt> right.
<audreyt> so, what to do with lib/pugs/run.pod ?
<audreyt> and hack.pod
<putter> oh, yeah, before I turn into a pumpkin, potentially netless for a few days, that's 
my main reaction to Deal,
<nothingmuch> so, no english volunteers?
<audreyt> can we merge both back to Pugs.pod (for now)?
<fglock> audreyt: Pugs::Doc(s)
<audreyt> Pugs::Doc::Run
<putter> Perl6 alpha will be mostly written in p6.  pugs+pil2js+pX+whatever.
<putter> That p6 mostly doesnt exist yet.
<putter> If it existed, it would actually not be tremendously difficult to do new backends, 
even from scratch.
<fglock> putter: not only backends, but compilers, optimizers
<putter> Now, when doing a backend, you have to simultaneously write that p6, which also 
means figuring out/disambiguating/making up a p6 spec.  Which all take far more time than 
actually doing the backend. ;)
<nothingmuch> you're all mean
<nothingmuch> you don't love me at all
<kolibrie> nothingmuch: we just don't think we speak English
* nothingmuch bops kolibrie
* kolibrie ducks
* nothingmuch kicks kolibrie while he's on the floor
<kolibrie> nothingmuch: url?
<fglock> nothingmuch: you mean - about the CV?
<nothingmuch> fglock: yes
<putter> fglock: right, not just backend.  parser, compiler, emitter, runtime, object space.  
the whole thing.
* kolibrie catches url offline
<svnbot6> r9366 | audreyt++ | * rename pugs::run and pugs::hack to Pugs::Doc::Run
<svnbot6> r9366 | audreyt++ |   and Pugs::Doc::Hack, for the Great Pugs Hierarchy.
<audreyt> fglock: okay. so move pX things over -- for non-perl6-pugs-specific things, like 
the iterator engine
<fglock> nothingmuch: I can help you with portuguese
<fglock> audreyt: which /dir?
<audreyt> you can still call it Pugs::Grammar and put into lib/Pugs/Grammar.pm
<audreyt> or if you want to distinguish components more
<nothingmuch> fglock: my sister might like help with her portuguese
<audreyt> lib/Pugs/Runtime/Grammar.pm
<audreyt> or if you are not unhappy with the Perl6 acmeness
<audreyt> then lib/Perl6/Grammar.pm
<putter> The drive spec schrodinger collapse with pugs and pil2js and pilrun has been great.  
but i suggest at this point, it's no longer the optimal path.  if we could get some folks 
simply nailing down the spec, that would free up implementation efforts to simply implement.  
<fglock> audreyt: ok!
<TimToady> oops.
<putter> nailing down spec == reading spec, making up all the questions you can think of, 
writing them down, getting them answered or approximated, and writing the answers.
<audreyt> (aka, ghost-writing)
<TimToady> Entirely coincidental, but I'm outta here.  (To see the dentist, again.)
<audreyt> rofl
<audreyt> TimToady: see ya :)
<TimToady> later, let's all have the appropriate amount of fun :/  &


-----------------

re: perl5 interop

<audreyt> { use Perl-6.0; ... write some perl code ... }
<audreyt> is the API I'd like
<fglock> audreyt: how about eval( '...', {lang=> 'Pugs' } )
<putter> (though it exists, and EngineReplace wouldnt exist without it, which forgives many 
sins)
<audreyt> it's even easy
<audreyt> just have a Perl.pm
<audreyt> with import that detects "-6.0"
<audreyt> or rather -6
<putter> yes, it is:)  # easy
* putter loves %^H.  which he knew about it years ago.
<putter> s/which/wishes/
<audreyt> it's extremely gonzo though, having Perl.pm -- it's like the ultimate violation of 
CPAN namespace guidelines :)
<putter> lol
<fglock> audreyt: isn't it 'use v6' ? --> 'v6.pm'
<putter> yes
<audreyt> no, that gets caught in the lexer
<audreyt> S11 has the
<audreyt> use Perl-6.0
<audreyt> form
<putter> really... oh, right
<audreyt> "use v6" is short for "use Perl-6" according to S11.
<putter> (i think the question was in perl5 space)
<audreyt> (I think we should have a p6 file that's both valid p5 and p6)
<audreyt> makes it easier to switch to pure-perl6 boot
<putter> http://www.vendian.org/mncharity/dir3/multilang/file/

<audreyt> so, in order to not upset the CPAN police
<putter> agreed
<audreyt> use Pugs::Perl-6.0; # maybe something like this.
<Juerd> audreyt: rafl and I had a half idea about maybe trying to organize an entire Perl 6 
track on froscon.
<putter> use Perl6 6-0; ?
<putter> err, 6.0

<Juerd> audreyt: As we all know that there's lots and lots to tell about Perl 6, pugs, 
parrot, etcetera.
<audreyt> Juerd: nod
<audreyt> putter:
<audreyt> use Perl6.0;
<audreyt> I think that works.

<audreyt> or, if we don't care about the CPAN police, we can go with "use Perl-6.0" -- which 
already exists on CPAN in version 0.03 ( 
http://search.cpan.org/~bmorrow/PerlInterp-0.03/Perl.pm )-- and will be usurped ;)
