chromatic: February 2008 Archives

The Perl 6 design team met by phone on 27 February 2008. Larry, Allison, Patrick, Will, Jerry, Jesse, Nicholas, and chromatic attended.

Larry:

• very busy with my day job right now
• clarified that multis and protos are automatically exported by default
• assumed that there's is export on them
• you have to declare them with my to get non-exported versions
• made some changes and defined something called interface versions
• subset of the version number
• the patchlevel doesn't matter when you require something with language tweaks
• only the interface version
• doing a lot of cleanup of STD.pm
• dekludging various things
• made a separate regular expression grammar
• derived from the Perl grammar
• allows us to override things like whitespace rules
• we use derivation for its original intent
• did a lot of work on the gimme program that translate that
• cleanups came from noticing badness when I tried to run the Perl 5
• plenty of feedback from Patrick, Jonathan, and the rest of the gang
• added some rules to STD.pm to add hooks for having an interactive parser that knows when you're done with user input
• the vws rule for vertical whitespace

Patrick:

• the Parrot release went very smoothly
• compliments to everyone who put together release notes and updated things for me
• spent the weekend in Brussels at FOSDEM 2008
• my Perl 6 talk seems to have been well received
• lots of compliments and comments and renewed enthusiasm
• spent some time on the plane looking into getting Rakudo to use Parrot's HLL directives
• it'll take some refactoring to get some of the classes working there
• just methods and the way things get stored in namespaces

Jerry:

• let's create a branch for that

Patrick:

• that's a good idea
• probably just a branch to play in, then switch over in the trunk
• probably doable incrementally
• did some work on trying to improve the state of Unicode in Parrot
• want to use more Unicode in Rakudo
• get string tests passing, use French quotes
• I'm a little concerned that applying the patch will slow things down until we can do things in something other than UTF-8
• otherwise applying patches and keeping up with the excellent work done by Jonathan, Jerry, and others
• all of the STD.pm changes have been really nice lately

Allison:

• working on the security PDD
• have a long list of features that we I include
• working on deciding a necessary and sufficient set of security features for the 1.0 release
• need the most useful features for our target contexts and target languages
• otherwise we could spend three years implementing security features

Jerry:

• enjoying some peanut chocolate-coated candies
• updated p6doc before the release
• I don't think anyone's really using it yet though
• removed some deprecated compiler directives from Rakudo's grammar
• now that we have fudge working
• working on a patch to allow you to write Rakudo runtime libraries in Perl 6
• Zev Benjamin sent the patch to the list
• it had some bugs, but most of those are fixed
• there seems to be one cross-compiler problem remaining
• made some progress on a YAML dumper to use for dumping AST match objects
• committed a config patch to detect gettext so we can work toward internationalizing Parrot

Jesse:

• talked to Mitchell Charity about the state of Pugs
• he's looking into a way to extract a standardized AST for all of the various Pugsian implementations for anyone to play with different backends
• a little glue around the kp6 AST to target from either end
• hopefully let the various folks in the Pugs community actually interoperate
• instead of having a half-dozen half-implemented full stacks hanging around

c:

• working on GC problems
• have a workaround I think fixes things
• sort of expensive, but we need to rethink part of our strategy
• workaround will probably stick around until we revise the GC in a couple of months
• did some profiling; think I'll tackle PIR profiling next

Nicholas:

• found a bug in Perl 1 configure and patched it

Jerry:

• do you have someone to bounce your security ideas off of?
• it would help me to have this more visible
• I don't want it to slow you down

Allison:

• this is just the initial draft
• the whole list will get to see and review the draft
• or do you want to see the mental process of what I consider and discard?

Jerry:

• I just wonder if you would have an easier time deciding what we need for 1.0 if you could discuss pros and cons with someone

Allison:

• I've been trying to write the PDD as a single coherent set of things
• haven't written pros and cons within the PDD
• otherwise it'll end up about 20 pages long
• so many different approaches for the same thing

Jerry:

• years ago, docs/dev/ was the place for those

Allison:

• I'll probably push the current document out today or tomorrow
• we'll start a series of discussions about various different topics
• a lot of it depends on what language implementors and application designers want
• the biggest thing I want is people asking for missing features and telling me they don't want other features
• what I'm doing is the core seed of that process

Jerry:

• hopefully interested people will read these minutes or a weblog post

Allison:

• I'll do my usual post to the mailing list

c:

• maybe you could post it to your journal

Allison:

• which journal?
• I've been thinking of setting up a dedicated Parrot blog

The Perl 6 design team met by phone on 20 February 2008. Larry, Allison, Patrick, Jerry, Will, Jesse, and chromatic attended.

Patrick:

• did a few small things in Rakudo over this week
• nothing spectacular or amazing
• will do the Parrot release later this evening
• expect no problems
• headed to FOSDEM in Belgium tomorrow to talk about Perl 6
• I should have some hacking time on airplanes
• very little chance of being interrupted

Larry:

• other than having the flu...
• doing various spec things
• documented what identifier extensions are
• straightened out the little mess in role composition as to the difference between role private attributes and the composed class's private attributes
• something declared with has is intended to be used generically within the class
• for a role private attribute, you declare it with my
• goes along with the use of my of giving private attributes (or methods at least) to classes
• if you declare unary or zero-arity functions with a prototype in Perl 5, that changes the grammar
• that's no longer the case in Perl 6
• they're still parsed as list operators
• you can define them that way if you want, but you have to do it explicitly

Patrick:

• that's a tremendous amount of sanity

Larry:

• some of us get there late

Patrick:

• some of us never get there

Larry:

• there's a big hole as to how backslash behaves in the interpolation spec
• what if you backslash something that isn't going to interpolate?
• the rules are similar to Perl 5, but more generically of course
• single-quotes assume you want to leave backslashes in
• double-quotes assume that all backslashes are meaningful, or they're in error
• working on how sig space interacts with ** on a separator
• fixing up some of the spec there
• came up in IRC
• the big thing this week is working on a program called gimme5
• chewing gum and bailing wire
• instead of spitting out Pugs code, it spits out Perl 5 code
• can now translate STD.pm to Perl 5 and run it under strict and warnings
• translated cursor.pm to Perl 5
• have the longest token scanner working for small, uncomplicated rules -- just alternatives
• need to revamp how longest-token matching works for more complicated rules
• the match is essentially over a list of alternatives
• given a rule that expects a term, it's zero or more prefix operators followed by a noun
• zero-or-more implies that the longest token at that point has to be the union of what prefix and noun matches
• it doesn't handle embedded alternations yet either
• the next step is doing what I could not make work under Pugs
• take the current match state and extract the match object to show to the user
• the current match state continuation has all of that information in it scattered among a bunch of backlinks
• it's not the form in which the user wants to access it
• unless the subscript lookups did some sort of tie-based behavior
• I don't want to do that
• we can just matchify things lazily when we know we need that for the user

Allison:

• finally done with my insane conference run
• no travel until the first week of March

c:

• doing a lot of work closing bugs
• working on open tickets
• closed a lot of them
• fixed a lot of little things for the release
• think I fixed one Tcl segfault
• working on the Lua GC bug now
• fixed up a lot of little things for the release

Jerry:

• doing more work on Rakudo
• found a weird bug that Patrick thought was GC, but it's not
• I did find a GC bug earlier today though and submitted a ticket
• did some other fixups for Parrot working on the release
• hope to work on the PDD 17 branch soon
• still have a Windows segfault there
• building on Linux so I can help
• started looking at gettext so we can internationalize Parrot
• I'll check something in for the config after the release
• after that, it's just a matter of putting in some macros and starting the conversion

Allison:

• the Linux kernel has some good hooks for i18n
• everything gets hidden behind a layer of abstraction, so you never directly print error text

Will:

• did a bunch of non-user-visible commits
• coding standard updates
• have a new contributor, Stephen Weeks
• he's contributed to several languages

Jerry:

• it feels like it's time to optimize PGE
• is that correct?

Patrick:

• there are lots of little improvements
• biggest improvement is probably the longest token match

c:

• can we avoid recursive descent with that?

Patrick:

• not entirely
• but we can avoid recursing into subrules that can't possibly match

c:

• I'm all for pruning trees

Patrick:

• longest token matching isn't trivial
• as Larry will tell you
• PGE needs to be able to attach attributes to subroutines
• I need to be able to ask a rule (or build a DFA) such that we can associate longest tokens with rules
• ask a rule "What does your DFA look like?"
• sounds like an attribute on a sub
• I haven't been able to put together a clean way to do that in Parrot
• we have subs with attributes or properties
• but once you take them down to being PIR
• you need a subroutine that attaches all of the properties once you load that subroutine

Larry:

• I'm not doing it that way
• I have an argument that's the state of the predetermination
• there's an artificial state, ?, that's "tell me your DFA"
• instead of caching that in the sub/method itself, there's a per-language cache
• if two different languages call into that rule, they can keep separate caches of their longest token matchers

Patrick:

• you have a way of calling a method that says "Give me your DFA" instead of "Do a match"?

Larry:

• yes
• you don't have to duplicate the dispatch
• you get the right one
• we don't try to duplicate the dispatch mechanism
• we just tell it to do something different than do the match

Patrick:

• the grammar holds that little bit of DFA
• keeps track of how they combine together

Larry:

• when you ask for the DFA for the particular rule, it asks its subrules
• each of those, the autolexer automatically knows whether it's calculated that
• it's already in the cache
• it reuses that bit then
• reincorporates that into a larger pattern
• off it goes
• the tricky thing is making sure that that cache is per-language
• Pugs doesn't do that now
• if you try to cache that in the sub as a property, you'd have to keep track of which language used it
• they have different lexers
• that's some of the motivation for simplifying the language tweaking things
• don't want the language to tweak for every unary or zero-array function
• I didn't want to cache duplicate lexers?

Patrick:

• part B of my answer is get some sort of profiling available in Parrot
• see what subs get called and where we spend our time
• part C is writing the Capture PMC in C
• that's waiting on the PMC changes

Jerry:

• we have a profile core in Parrot, -p

Will:

• it's based on the C level

Jerry:

• you want subroutines, not ops?

Patrick:

• yes
• making a native Capture PMC could be a big win
• it just redispatches to arrays and hashes
• that deserves to be its own beast

Larry:

• do you maintain that match object on the fly as a mutable object, and back things out as you backtrack?
• my passed-around match state is cloned, immutable objects
• my continuation system just uses the cloned snapshot
• it may or may not be faster
• I know where I need to reconstruct that mutable match object that the user sees
• that's another possible optimization/pessimization

Patrick:

• the match object is a convenient place to keep track of backtrack state
• backing out is basically a delete optimization
• I could optimize the backtracking
• function is more important than speed for what I've done so far
• another big win is getting strings implemented under Parrot such that we're not using UTF-8
• that's slow
• because of ICU and Unicode support, we're stuck with UTF-8 at the moment

Larry:

• I've done enough of that in my life

Patrick:

• as far as the code it generates, I'm sure people can come up with improvements there
• the code itself does a lot of optimizations
• profiling seems to be the big win

c:

• I have some ideas there

The Perl 6 design team met by phone on 13 February 2008. Larry, Allison, Jerry, Will, Richard, Patrick, and chromatic attended.

c:

• fixed a few more bugs
• haven't made a lot of progress this week
• working through the CAGE tasks Will gave me
• talked with Allison about the garbage collector
• going to ask Coverity about resuming their scans

Larry:

• continue to work on the item/list assignment semantics in F
• pretty much have the longest-token autolexer working, using tagged regular expressions
• it takes about 10 CPU minutes running Pugs to calculate an autolexer for a noun
• I can speed that up greatly by caching
• mostly that's because Pugs is very slow
• thought about the relationship of autothreading with only subs
• multi subs use multiple dispatch naturally to autothread on Junctions
• but only subs are not multiply dispatched by definition
• figuring out a way to do that without putting the overhead of multiple dispatch on normal calls
• there's no Undef type anymore
• there are many varieties of Undef, such as Object or Failure
• the undef keyword remains; it returns the most general type of Failure
• S03 doesn't talk about associativity much, so I specced that for most operators
• did some thinking about modules that do language tweaks
• I think we should do strict versioning of modules in the official library when they do language tweaks
• you have to specify the exact version number when you pull one in
• we know exactly which language you get then
• this is for the benefit of tool writers
• a tool can know which version of the language it has by the version without having to pull in the module
• this requirement is much looser for your own modules; mostly applies by default to the standard library
• watching putter do a hand-translation of the standard grammar to Ruby
• he's finding some things to patch
• continuing to stay out of my own thread on pluralization :)

Patrick:

• spent the last week in the hospital with Paula
• nothing's wrong, but they just wouldn't let her go home
• catching up on email the past couple of days
• responded to the lexicals question
• writing a short report on the grant progress in the next couple of days
• hope to be back working on that this week
• need to review the changes to the system in detail from the past few weeks
• going to postpone the release from next Tuesday to next Wednesday

Jerry:

• had a couple of conversations with Larry and ruoso about SMOP

Allison:

• spent the weekend at the SCALE conference
• made some interesting contacts for Parrot and TPF funding
• one more bug in the PMC parser for attributes
• it parses and generates the attribute structs fine, but fails to parse the rest of the file
• working on the security PDD; due in a few days
• talked to Chris Shiflett at Foo Camp
• he had some good ideas
• Rasmus had some good ideas for security hooks too

c:

• David Fetter is a good contact too

Allison:

• I'll be at Frozen Perl this weekend
• the hallway development track mostly

Will:

• trying to keep on track of RT tickets
• assigned a few to chromatic
• trying to track the progress of the MacPorts for the 0.5.2 release
• appears stalled; can't seem to find the original maintainer
• may be able to stake my claim in the next week

Richard:

• spent a few days at the MoneyTech conference
• wanted to see how deeply the finance world is into Perl
• made some efforts to start things up with regard to sponsorship
• talked to a guy from Intel who was there to talk about parallelization
• working on a followup conversation there
• looking into upcoming conferences: EclipseCon, LSB conference

Patrick:

• would that copying GC scheme help us with the copy opcode?
• if you have a way to collapse reference transparently, you can pass through access to the old PMC to the new PMC

Allison:

• we haven't decided whether that copying scheme is a pre 1.0 or post 1.0 feature

c:

• seems like 1.2 to me

The Perl 6 design team met by phone on 06 February 2008. Larry, Jerry, Will, Jesse, Nicholas, and chromatic attended.

c:

• fixed a couple of bugs in Parrot
• still have a few more to investigate (hopefully will get to Tcl and Lua)
• spent a lot of time fixing memory leaks in IMCC
• lots of refactoring there with Andy, lots to go
• hopeful we can explore a switch to PIRC in the near future
• a little bit of work left on the PMC PDD
• promised to review the GC PDD; some concerns there

Larry:

• spent most of the week contemplating and implementing changes to the way assignment decides whether it's item or list assignment
• the old method was quite insane in various ways
• if you're assigning to a scalar, it's an item
• assignment otherwise parses as list assignment
• users and parsers should be able to keep easy track of
• also actually edited Pugs, in Haskell, to change the determination semantics
• it actually recompiled successfully

Nicholas:

• I didn't know it compiled at all these days

Larry:

• I'm still on GHC 6.6.1, and that works
• someone proposed named placeholders
• very handy for certain database problems
• there are various contexts where that would be useful
• you could do that by dereferencing the same hash...
• because $:foo is still available and :foo looks like a named variable, I made that the named version of $^a

Jesse:

• how similar does that end up looking to named captures?
• is the syntax similar?
• in my head, named captures and named placeholders...

Larry:

• named captures end up in a hash
• never get bound to variables
• this actually creates variables
• also if you refer to @_ or %_ in one of these subs, it automatically adds that as a slurpy on the end
• you can get control of unrecognized placeholders there
• for a better error message, for example
• just filling out named placeholders
• not sure how generally useful it will be
• it felt more orthogonal for me
• created another twigil
• filling out the twigil set... the more twigils we have, the more it justifies the twigil design
• not a very strong razor, but it's a razor
• it doesn't interfere with the other colon declarative "make this a pair" argument in signatures
• hanging out, answering questions, tweaking F where it's unclear or people notice a fix
• putter's working on yet another F analyzer
• intended for use in backends other than Pugs
• whether we use that for anything, he's discovering various infelicities in the standard grammar
• TPF turned down ruoso for a grant to work on SMOP
• discouraged him some
• his work has helped me understand the ramifications of things like captures at a very fundamental level

Jerry:

• worked with Andy a bit this week to wrap snprintf() and get it working on Windows
• it's not C89
• glad to see Larry's work on list and item assignment
• looking forward to implementing that with Patrick
• submitted OSCON talks for Patrick and myself
• falling behind on development due to real life
• hope to pick up next week sometime
• I don't want the pace to slow too much, with Patrick unavailable this week
• also committed a first pass at perl6doc for Rakudo this week
• it's not much, but it's a place where people can start hacking on the POD grammar
• I know people have held back from implementing things because the spec isn't finished
• I don't see any point in waiting for implementing things here

Will:

• I agree
• we'll have to cut a release at some point
• #parrot is now logged
• hopefully that'll help people get up to speed
• going through and trying to clean up some old RT tickets
• trying to get movement on things that are a simple matter of cleanup
• oh, and I'm on TPF's grant committee now as a Perl 6 person

Nicholas:

• brian d foy raised a bug about the issue of when with hashes and arrays in 5.10
• he's trying to teach the stuff and work out examples
• I haven't looked at it, because I'm focusing on 5.8.9