February 2008 Archives

The Perl 6 design team met by phone on 27 February 2008. Larry, Allison, Patrick, Will, Jerry, Jesse, Nicholas, and chromatic attended.

Larry:

• very busy with my day job right now
• clarified that multis and protos are automatically exported by default
• assumed that there's is export on them
• you have to declare them with my to get non-exported versions
• made some changes and defined something called interface versions
• subset of the version number
• the patchlevel doesn't matter when you require something with language tweaks
• only the interface version
• doing a lot of cleanup of STD.pm
• dekludging various things
• made a separate regular expression grammar
• derived from the Perl grammar
• allows us to override things like whitespace rules
• we use derivation for its original intent
• did a lot of work on the gimme program that translate that
• cleanups came from noticing badness when I tried to run the Perl 5
• plenty of feedback from Patrick, Jonathan, and the rest of the gang
• added some rules to STD.pm to add hooks for having an interactive parser that knows when you're done with user input
• the vws rule for vertical whitespace

Patrick:

• the Parrot release went very smoothly
• compliments to everyone who put together release notes and updated things for me
• spent the weekend in Brussels at FOSDEM 2008
• my Perl 6 talk seems to have been well received
• lots of compliments and comments and renewed enthusiasm
• spent some time on the plane looking into getting Rakudo to use Parrot's HLL directives
• it'll take some refactoring to get some of the classes working there
• just methods and the way things get stored in namespaces

Jerry:

• let's create a branch for that

Patrick:

• that's a good idea
• probably just a branch to play in, then switch over in the trunk
• probably doable incrementally
• did some work on trying to improve the state of Unicode in Parrot
• want to use more Unicode in Rakudo
• get string tests passing, use French quotes
• I'm a little concerned that applying the patch will slow things down until we can do things in something other than UTF-8
• otherwise applying patches and keeping up with the excellent work done by Jonathan, Jerry, and others
• all of the STD.pm changes have been really nice lately

Allison:

• working on the security PDD
• have a long list of features that we I include
• working on deciding a necessary and sufficient set of security features for the 1.0 release
• need the most useful features for our target contexts and target languages
• otherwise we could spend three years implementing security features

Jerry:

• enjoying some peanut chocolate-coated candies
• updated p6doc before the release
• I don't think anyone's really using it yet though
• removed some deprecated compiler directives from Rakudo's grammar
• now that we have fudge working
• working on a patch to allow you to write Rakudo runtime libraries in Perl 6
• Zev Benjamin sent the patch to the list
• it had some bugs, but most of those are fixed
• there seems to be one cross-compiler problem remaining
• made some progress on a YAML dumper to use for dumping AST match objects
• committed a config patch to detect gettext so we can work toward internationalizing Parrot

Jesse:

• talked to Mitchell Charity about the state of Pugs
• he's looking into a way to extract a standardized AST for all of the various Pugsian implementations for anyone to play with different backends
• a little glue around the kp6 AST to target from either end
• hopefully let the various folks in the Pugs community actually interoperate
• instead of having a half-dozen half-implemented full stacks hanging around

c:

• working on GC problems
• have a workaround I think fixes things
• sort of expensive, but we need to rethink part of our strategy
• workaround will probably stick around until we revise the GC in a couple of months
• did some profiling; think I'll tackle PIR profiling next

Nicholas:

• found a bug in Perl 1 configure and patched it

Jerry:

• do you have someone to bounce your security ideas off of?
• it would help me to have this more visible
• I don't want it to slow you down

Allison:

• this is just the initial draft
• the whole list will get to see and review the draft
• or do you want to see the mental process of what I consider and discard?

Jerry:

• I just wonder if you would have an easier time deciding what we need for 1.0 if you could discuss pros and cons with someone

Allison:

• I've been trying to write the PDD as a single coherent set of things
• haven't written pros and cons within the PDD
• otherwise it'll end up about 20 pages long
• so many different approaches for the same thing

Jerry:

• years ago, docs/dev/ was the place for those

Allison:

• I'll probably push the current document out today or tomorrow
• we'll start a series of discussions about various different topics
• a lot of it depends on what language implementors and application designers want
• the biggest thing I want is people asking for missing features and telling me they don't want other features
• what I'm doing is the core seed of that process

Jerry:

• hopefully interested people will read these minutes or a weblog post

Allison:

• I'll do my usual post to the mailing list

c:

• maybe you could post it to your journal

Allison:

• which journal?
• I've been thinking of setting up a dedicated Parrot blog

The Perl 6 design team met by phone on 20 February 2008. Larry, Allison, Patrick, Jerry, Will, Jesse, and chromatic attended.

Patrick:

• did a few small things in Rakudo over this week
• nothing spectacular or amazing
• will do the Parrot release later this evening
• expect no problems
• headed to FOSDEM in Belgium tomorrow to talk about Perl 6
• I should have some hacking time on airplanes
• very little chance of being interrupted

Larry:

• other than having the flu...
• doing various spec things
• documented what identifier extensions are
• straightened out the little mess in role composition as to the difference between role private attributes and the composed class's private attributes
• something declared with has is intended to be used generically within the class
• for a role private attribute, you declare it with my
• goes along with the use of my of giving private attributes (or methods at least) to classes
• if you declare unary or zero-arity functions with a prototype in Perl 5, that changes the grammar
• that's no longer the case in Perl 6
• they're still parsed as list operators
• you can define them that way if you want, but you have to do it explicitly

Patrick:

• that's a tremendous amount of sanity

Larry:

• some of us get there late

Patrick:

• some of us never get there

Larry:

• there's a big hole as to how backslash behaves in the interpolation spec
• what if you backslash something that isn't going to interpolate?
• the rules are similar to Perl 5, but more generically of course
• single-quotes assume you want to leave backslashes in
• double-quotes assume that all backslashes are meaningful, or they're in error
• working on how sig space interacts with ** on a separator
• fixing up some of the spec there
• came up in IRC
• the big thing this week is working on a program called gimme5
• chewing gum and bailing wire
• instead of spitting out Pugs code, it spits out Perl 5 code
• can now translate STD.pm to Perl 5 and run it under strict and warnings
• translated cursor.pm to Perl 5
• have the longest token scanner working for small, uncomplicated rules -- just alternatives
• need to revamp how longest-token matching works for more complicated rules
• the match is essentially over a list of alternatives
• given a rule that expects a term, it's zero or more prefix operators followed by a noun
• zero-or-more implies that the longest token at that point has to be the union of what prefix and noun matches
• it doesn't handle embedded alternations yet either
• the next step is doing what I could not make work under Pugs
• take the current match state and extract the match object to show to the user
• the current match state continuation has all of that information in it scattered among a bunch of backlinks
• it's not the form in which the user wants to access it
• unless the subscript lookups did some sort of tie-based behavior
• I don't want to do that
• we can just matchify things lazily when we know we need that for the user

Allison:

• finally done with my insane conference run
• no travel until the first week of March

c:

• doing a lot of work closing bugs
• working on open tickets
• closed a lot of them
• fixed a lot of little things for the release
• think I fixed one Tcl segfault
• working on the Lua GC bug now
• fixed up a lot of little things for the release

Jerry:

• doing more work on Rakudo
• found a weird bug that Patrick thought was GC, but it's not
• I did find a GC bug earlier today though and submitted a ticket
• did some other fixups for Parrot working on the release
• hope to work on the PDD 17 branch soon
• still have a Windows segfault there
• building on Linux so I can help
• started looking at gettext so we can internationalize Parrot
• I'll check something in for the config after the release
• after that, it's just a matter of putting in some macros and starting the conversion

Allison:

• the Linux kernel has some good hooks for i18n
• everything gets hidden behind a layer of abstraction, so you never directly print error text

Will:

• did a bunch of non-user-visible commits
• coding standard updates
• have a new contributor, Stephen Weeks
• he's contributed to several languages

Jerry:

• it feels like it's time to optimize PGE
• is that correct?

Patrick:

• there are lots of little improvements
• biggest improvement is probably the longest token match

c:

• can we avoid recursive descent with that?

Patrick:

• not entirely
• but we can avoid recursing into subrules that can't possibly match

c:

• I'm all for pruning trees

Patrick:

• longest token matching isn't trivial
• as Larry will tell you
• PGE needs to be able to attach attributes to subroutines
• I need to be able to ask a rule (or build a DFA) such that we can associate longest tokens with rules
• ask a rule "What does your DFA look like?"
• sounds like an attribute on a sub
• I haven't been able to put together a clean way to do that in Parrot
• we have subs with attributes or properties
• but once you take them down to being PIR
• you need a subroutine that attaches all of the properties once you load that subroutine

Larry:

• I'm not doing it that way
• I have an argument that's the state of the predetermination
• there's an artificial state, ?, that's "tell me your DFA"
• instead of caching that in the sub/method itself, there's a per-language cache
• if two different languages call into that rule, they can keep separate caches of their longest token matchers

Patrick:

• you have a way of calling a method that says "Give me your DFA" instead of "Do a match"?

Larry:

• yes
• you don't have to duplicate the dispatch
• you get the right one
• we don't try to duplicate the dispatch mechanism
• we just tell it to do something different than do the match

Patrick:

• the grammar holds that little bit of DFA
• keeps track of how they combine together

Larry:

• when you ask for the DFA for the particular rule, it asks its subrules
• each of those, the autolexer automatically knows whether it's calculated that
• it's already in the cache
• it reuses that bit then
• reincorporates that into a larger pattern
• off it goes
• the tricky thing is making sure that that cache is per-language
• Pugs doesn't do that now
• if you try to cache that in the sub as a property, you'd have to keep track of which language used it
• they have different lexers
• that's some of the motivation for simplifying the language tweaking things
• don't want the language to tweak for every unary or zero-array function
• I didn't want to cache duplicate lexers?

Patrick:

• part B of my answer is get some sort of profiling available in Parrot
• see what subs get called and where we spend our time
• part C is writing the Capture PMC in C
• that's waiting on the PMC changes

Jerry:

• we have a profile core in Parrot, -p

Will:

• it's based on the C level

Jerry:

• you want subroutines, not ops?

Patrick:

• yes
• making a native Capture PMC could be a big win
• it just redispatches to arrays and hashes
• that deserves to be its own beast

Larry:

• do you maintain that match object on the fly as a mutable object, and back things out as you backtrack?
• my passed-around match state is cloned, immutable objects
• my continuation system just uses the cloned snapshot
• it may or may not be faster
• I know where I need to reconstruct that mutable match object that the user sees
• that's another possible optimization/pessimization

Patrick:

• the match object is a convenient place to keep track of backtrack state
• backing out is basically a delete optimization
• I could optimize the backtracking
• function is more important than speed for what I've done so far
• another big win is getting strings implemented under Parrot such that we're not using UTF-8
• that's slow
• because of ICU and Unicode support, we're stuck with UTF-8 at the moment

Larry:

• I've done enough of that in my life

Patrick:

• as far as the code it generates, I'm sure people can come up with improvements there
• the code itself does a lot of optimizations
• profiling seems to be the big win

c:

• I have some ideas there

I've been at the German Perl Workshop for the last few days. While I was sat in a bunch of talks being presented in a language I don't understand, I had some free time to hack on Rakudo. That means there's been a few fixes and some new features.

First, I set about doing some re-factoring to the objects work I had done so far. It had wound up with a lot of inline PIR in actions.pm, and that in turn fixed us to one class system, which was fine for a first-cut implementation to get a few things working, but far from what S12 calls for. Now all the inline PIR I added is gone, replaced by calls to methods written in PIR.

With the refactoring done, it was then far easier to add what is needed to have methods in classes resolving conflicts and taking precedence during role composition. Of course, exactly how well the roles implementation matches S12 is yet to be seen, and that calls for lots of tests. I'll be drawing on the Moose ones for that.

With some basic OO support in and ready for people to play with, I moved on to looking at the regex support. There was already some there, but I've put in a bit more. $/ (the match variable) wasn't being passed into inner scopes, so if you did a pattern match in the condition of an if block then $/ would not be set inside the block. That is now fixed. Additionally, I implemented $0, $1, $2, etc numbered captures, as well as $ and $ named captures. There is also the "regex" keyword for introducing named regexes, so you can say stuff like:

regex Year {\d\d\d\d};
regex Location {German|French|Italian|London|Dutch|Ukrainian};
regex PerlConference {<Location>\sPerl\sWorkshop[\s<Year>]?};


I've got rule and token parsing, but they don't pass along the :ratchet and :sigspace modifiers yet. I hope to have that resolved soon.

A natural fall-out of adding support for $ is that %hash is now also supported for using constant hash keys. So that's another small bit done. Anyway, enjoy the new toys, and bug reports welcome.

The Perl 6 design team met by phone on 13 February 2008. Larry, Allison, Jerry, Will, Richard, Patrick, and chromatic attended.

c:

• fixed a few more bugs
• haven't made a lot of progress this week
• working through the CAGE tasks Will gave me
• talked with Allison about the garbage collector
• going to ask Coverity about resuming their scans

Larry:

• continue to work on the item/list assignment semantics in F
• pretty much have the longest-token autolexer working, using tagged regular expressions
• it takes about 10 CPU minutes running Pugs to calculate an autolexer for a noun
• I can speed that up greatly by caching
• mostly that's because Pugs is very slow
• thought about the relationship of autothreading with only subs
• multi subs use multiple dispatch naturally to autothread on Junctions
• but only subs are not multiply dispatched by definition
• figuring out a way to do that without putting the overhead of multiple dispatch on normal calls
• there's no Undef type anymore
• there are many varieties of Undef, such as Object or Failure
• the undef keyword remains; it returns the most general type of Failure
• S03 doesn't talk about associativity much, so I specced that for most operators
• did some thinking about modules that do language tweaks
• I think we should do strict versioning of modules in the official library when they do language tweaks
• you have to specify the exact version number when you pull one in
• we know exactly which language you get then
• this is for the benefit of tool writers
• a tool can know which version of the language it has by the version without having to pull in the module
• this requirement is much looser for your own modules; mostly applies by default to the standard library
• watching putter do a hand-translation of the standard grammar to Ruby
• he's finding some things to patch
• continuing to stay out of my own thread on pluralization :)

Patrick:

• spent the last week in the hospital with Paula
• nothing's wrong, but they just wouldn't let her go home
• catching up on email the past couple of days
• responded to the lexicals question
• writing a short report on the grant progress in the next couple of days
• hope to be back working on that this week
• need to review the changes to the system in detail from the past few weeks
• going to postpone the release from next Tuesday to next Wednesday

Jerry:

• had a couple of conversations with Larry and ruoso about SMOP

Allison:

• spent the weekend at the SCALE conference
• made some interesting contacts for Parrot and TPF funding
• one more bug in the PMC parser for attributes
• it parses and generates the attribute structs fine, but fails to parse the rest of the file
• working on the security PDD; due in a few days
• talked to Chris Shiflett at Foo Camp
• he had some good ideas
• Rasmus had some good ideas for security hooks too

c:

• David Fetter is a good contact too

Allison:

• I'll be at Frozen Perl this weekend
• the hallway development track mostly

Will:

• trying to keep on track of RT tickets
• assigned a few to chromatic
• trying to track the progress of the MacPorts for the 0.5.2 release
• appears stalled; can't seem to find the original maintainer
• may be able to stake my claim in the next week

Richard:

• spent a few days at the MoneyTech conference
• wanted to see how deeply the finance world is into Perl
• made some efforts to start things up with regard to sponsorship
• talked to a guy from Intel who was there to talk about parallelization
• working on a followup conversation there
• looking into upcoming conferences: EclipseCon, LSB conference

Patrick:

• would that copying GC scheme help us with the copy opcode?
• if you have a way to collapse reference transparently, you can pass through access to the old PMC to the new PMC

Allison:

• we haven't decided whether that copying scheme is a pre 1.0 or post 1.0 feature

c:

• seems like 1.2 to me

Recently I've been doing more work on implementing Perl 6 OO support in Rakudo. You can read the full write-up, but as an overview:

• Attributes are now more in line with S12, storing them at $!foo under the hood no matter what twigil they are declared with, generating an accessor/mutator of the twigil is '.' and creating a lexical alias to the attribute if there is no twigil (so has $x means you can use $x and $!x to refer to the attribute).
• The initial work to support inheritance is in place. And it really is implemented through a generic trait application mechanism, not special-case code.
• The "self" keyword can be used now.
• The very initial bits of work on roles are in, but the composition semantics are not correct yet. That will change soon.

Enjoy, and I hope to have more stuff to report on soon.

The Perl 6 design team met by phone on 06 February 2008. Larry, Jerry, Will, Jesse, Nicholas, and chromatic attended.

c:

• fixed a couple of bugs in Parrot
• still have a few more to investigate (hopefully will get to Tcl and Lua)
• spent a lot of time fixing memory leaks in IMCC
• lots of refactoring there with Andy, lots to go
• hopeful we can explore a switch to PIRC in the near future
• a little bit of work left on the PMC PDD
• promised to review the GC PDD; some concerns there

Larry:

• spent most of the week contemplating and implementing changes to the way assignment decides whether it's item or list assignment
• the old method was quite insane in various ways
• if you're assigning to a scalar, it's an item
• assignment otherwise parses as list assignment
• users and parsers should be able to keep easy track of
• also actually edited Pugs, in Haskell, to change the determination semantics
• it actually recompiled successfully

Nicholas:

• I didn't know it compiled at all these days

Larry:

• I'm still on GHC 6.6.1, and that works
• someone proposed named placeholders
• very handy for certain database problems
• there are various contexts where that would be useful
• you could do that by dereferencing the same hash...
• because $:foo is still available and :foo looks like a named variable, I made that the named version of $^a

Jesse:

• how similar does that end up looking to named captures?
• is the syntax similar?
• in my head, named captures and named placeholders...

Larry:

• named captures end up in a hash
• never get bound to variables
• this actually creates variables
• also if you refer to @_ or %_ in one of these subs, it automatically adds that as a slurpy on the end
• you can get control of unrecognized placeholders there
• for a better error message, for example
• just filling out named placeholders
• not sure how generally useful it will be
• it felt more orthogonal for me
• created another twigil
• filling out the twigil set... the more twigils we have, the more it justifies the twigil design
• not a very strong razor, but it's a razor
• it doesn't interfere with the other colon declarative "make this a pair" argument in signatures
• hanging out, answering questions, tweaking F where it's unclear or people notice a fix
• putter's working on yet another F analyzer
• intended for use in backends other than Pugs
• whether we use that for anything, he's discovering various infelicities in the standard grammar
• TPF turned down ruoso for a grant to work on SMOP
• discouraged him some
• his work has helped me understand the ramifications of things like captures at a very fundamental level

Jerry:

• worked with Andy a bit this week to wrap snprintf() and get it working on Windows
• it's not C89
• glad to see Larry's work on list and item assignment
• looking forward to implementing that with Patrick
• submitted OSCON talks for Patrick and myself
• falling behind on development due to real life
• hope to pick up next week sometime
• I don't want the pace to slow too much, with Patrick unavailable this week
• also committed a first pass at perl6doc for Rakudo this week
• it's not much, but it's a place where people can start hacking on the POD grammar
• I know people have held back from implementing things because the spec isn't finished
• I don't see any point in waiting for implementing things here

Will:

• I agree
• we'll have to cut a release at some point
• #parrot is now logged
• hopefully that'll help people get up to speed
• going through and trying to clean up some old RT tickets
• trying to get movement on things that are a simple matter of cleanup
• oh, and I'm on TPF's grant committee now as a Perl 6 person

Nicholas:

• brian d foy raised a bug about the issue of when with hashes and arrays in 5.10
• he's trying to teach the stuff and work out examples
• I haven't looked at it, because I'm focusing on 5.8.9

I strongly believe Rakudo Perl will have faster adoption if it feels like Perl. I've recently added a simple perl6doc utility to Rakudo Perl to move us one small step in that direction. Check out my journal post for more details, and stay tuned for updates to perl6doc and more.