Rakudo.org

The Perl 6 design team met by phone on 07 May 2008. Larry, Allison, Patrick, Will, Jerry, Nicholas, Jesse, and chromatic attended.

Allison:

• spent my time this week slicing and dicing the exceptions implementation
• replaced the old internals with the new system
• checked that in yesterday
• still a few failing tests in edge cases on the branch
• did more work on the Parrot Foundation

c:

• I own an acre of Mars, we could incorporate there

Allison:

• don't you own a cow in the Philippines?

c:

• yes, but that doesn't give me any governmental powers

Patrick:

• isn't that worth a lot?

c:

• the peso is improving against the dollar

Jesse:

• moving on...

Larry:

• clear bill of health from my medical reports
• hacking a lot against the standard grammar in my STD5 implementation
• lots of refactoring
• all of the various parameters that used to go through separately now go through as part of the Match object
• including the "fate" and whether we're peeking at the longest token set
• the longest token matcher works now
• I threw out my old mechanism for gathering Match objects
• it now creates the more-or-less correctly
• lots of grammar tweaks, as suggested by Mitchell Charity
• lots of refactoring of how logging works so that it doesn't always spew enormous quantities of information to the screen
• I can actually run the parser quite quickly now, for some definition of quickly that approximates 2000 characters per second
• matches symbols directly, rather than calling a rule, which is faster
• does the backoff now on longest token matching
• started refactoring the grammar on the assumption that I cna trust the longest token matcher
• no longer any nofat rule
• the longest token should match the fat arrow, if there is one
• started refactoring the quoting rules to parse as if they were sublanguages
• getting rid of extra rigamarole to recreate the other mechanism we already use for other languages
• working out the linkage for switching in and out of sublanguages
• how to get to the outer language from the inner language
• calling into pure Perl from closures in a regex
• or the host language if you're calling the regex from another language
• nailed down the available methods for Match objects in the specs
• giving a talk in Seattle on Friday at SPU
• flying to Japan on Saturday

Patrick:

• spent a lot of time teaching this past week
• cleared up now
• mostly I've answered questions on mailing lists and IRC
• I'm not always sure that I'm helpful, but I'm there
• yesterday I worked on trying to get a bunch of little small things here and there
• fixed up a few things in PCT internals
• today I'm bringing PGE up to date with some of the latest changes in S05
• these all help Rakudo and other languages in small ways
• trying to clean out my backlog and clean up a bunch of RT tickets
• I'll continue over the next couple of days
• and blogging about it as I go

Jerry:

• things are busy, mostly non-Parrot related stuff
• submitted a ticket that I hope Patrick can close today

Patrick:

• many languages depend on the old behavior, including Plumhead
• I'm not certain about some of them

Jerry:

• mostly otherwise answering questions on #parrot
• making sure that things are set up for the real work phase of GSoC
• making sure that students have their CLAs, if not commit bits
• astonished to see how much work Jonathan is getting done in just two funded days
• it's amazing to see how much a motivator money can be
• I'd like to see more of it, hint hint

c:

• working on closing as many open Parrot bugs as possible
• applying as many open patches as possible
• should be able to help on the concurrency branch soon
• otherwise preparing for the release
• going to check on received CLAs this week

Nicholas:

• found it curious that Perl 5.10 has the best state implementation of any language
• wanted to steal tests from another implementation
• had a discussion with Leon about SMOP
• there's no real description of how all of these implementations fit together
• Rakudo plus Parrot is a complete implementation
• SMOP and kp6 fit together nicely

Jesse:

• I started a wiki page on the Perl 6 wiki at Perl_6_Implementations

Patrick:

• I don't know that it says how things fit together

Jesse:

• I tried to encourage other people to contribute stuff
• didn't get much uptake

Nicholas:

• should we suggest to Daniel that he should help explain things?

Jesse:

• that's more likely to get people contributing to it

Will:

• there's definitely some confusion about it within the Grants Committee

Jerry:

• SMOP has the highest documentation-to-line-of-code ratio of any implementation

Patrick:

• it needs a good overview though

Nicholas:

• I'll ask Daniel to explain more
• especially its relationship to Parrot and Rakudo

Jerry:

• it sounds like it could be an alternate runcore for Perl 5 as well

Jesse:

• tried a few different things
• decided to write a test for Rakudo
• tried a simple arithmetic test pulled from Pugs
• found that Rakudo didn't implement a function specified in the S29 draft
• Patrick helped me write a couple of lines of code to implement it
• then discovered that fudge didn't support try blocks in a specific way
• Larry patched that
• then found that incrementing an undefined value didn't work in Rakudo
• that was the end of my day
• I still need to write up my findings
• how easy is it for someone without experience in Rakudo and its internals to pick things up and contribute something?
• more difficult than I thought it might be, but it's getting more doable
• it's important to understand how it might fail before trying to get people to do it
• then I started trying to play with MAD on the weekend
• found and fixed a bug in its XML
• refactored it such that you can run MAD's tests in the core if you add a copy of XML::Parser to the core
• it's not far enough yet, but it's a start

Nicholas:

• is it going to be difficult to restructure the Parrot foundation from 501(c)(3) to 501(c)(6)?

Allison:

• you can do pretty much the same thing
• sponsors are on the board in a c6
• they're only advisory in a c3
• the sponsors we've talked to are mostly only interested in getting regular status reports and the like

Jesse:

• is there any jumping around to transfer copyright to the new foundation?

Allison:

• we'll do a copyright assignment from the Perl Foundation to the Parrot Foundation
• all of the CLAs that went into the code up to the point of signover will be fine
• but we'll essentially copy the Perl Foundation CLA to a Parrot Foundation CLA

Will:

• do we need to contact committers who haven't signed a CLA?

Patrick:

• where does Rakudo fall?

Allison:

• still under the Perl Foundation
• it doesn't move at all

Will:

• do we want to split up the repository at that point?

Allison:

• eventually, we'll want to do so anyway
• it's not an urgent thing

Jerry:

• what would it take to version a Perl6Regex frontend to PGE?
• let grammars specify a version of the grammar

Patrick:

• I did that before by having a separate compiler
• you're talking about something a bit finer grained
• I don't want to do that
• as we get closer to 1.0, that'd be fine
• I already have enough to do keeping up with the latest versions

Will:

• I don't think we want to keep up old versions

Patrick:

• I don't mind sticking to our deprecation cycle
• I hadn't put the change from today into the deprecation list yet
• we'll get to it in a couple of weeks

Jerry:

• just trying to figure out how to push forward with changes to PGE without having to update every language in the repository

Patrick:

• freeze S05?
• not a great solution

Larry:

• I heard that

Patrick:

• the last few changes have been great
• I'm not really serious about that

Larry:

• some of them you even asked for

c:

• it's an advantage to have these languages in the repository
• we can update them
• but only if we can run the tests before and after and know that they pass

Will:

• we might consider removing languages with failing tests and no recent updates
• there are 17 grant proposals, some of them Perl 6-related
• please comment on the TPF blog
• it'll help

Jesse:

• blog.perlfoundation.org

I started out the day by looking through the RT queue for Rakudo. Two tickets were already dealt with, so I just closed those. Another was a bug report concerning assigning undef to typed variables. Doing:

my Int $x = undef;


Would give a type check failure. This is now resolved. Furthermore, if your type is a class name, then assigning undef at any point to it will result in it holding the protoobject for that class again. I also took a moment to post to Perl 6 Language to get some clarifications on what "my TypeName $x;" left $x being when $x was a role, subset type or a junction of types.

Last week I started getting grammars in place. I got so far as having the regex live in the correct namespaces, but that didn't make grammars at all class-like, which is how they should be. This week I set out to fix that. Grammars now get protoobjects too, which you can call .WHAT and so forth on. Furthermore, I got inheritance working and also smart-matching against a grammar, which runs it's TOP rule. Therefore you can now run the following example in Rakudo.

grammar Loads { regex Lots { \d+s } };
grammar Many is Loads { rule TOP { of } };
if "100s of 1000s" ~~ Many {
    say $/; # 100s of 1000s
}


Having closed four RT tickets so far, I took a look through there to see what else there was. There was one that did most of what was needed to implement the .perl method on Junctions (which you can call, in theory anyway, on anything to get a Perl representation of it). I did the required fixes and applied it, but realized in the process that we were missing .perl for any of the really fundamental types, so I added it for Num, Int and Str.

With that done, I spent a little time on the S12 tests. I added fudge directives to get one of the that failed to parse to do so, and added an extra test. I plan to add much more here and flesh out the tests quite a bit over time.

Turning back to the OO support, I did some updates to the grammar that both brought us closer to STD.pm - the official grammar - and added the ability to parse a range of extra things. The first grammar changes were related to method calling. Normally you call a method just with ".", but private methods are called with "!". Additionally, there are ways to call sets of methods with quantifiers (.?, .+ and .*, with meanings analogous to those in regexes). Finally, I added the ability to scope declarations of routines too, so we can now parse lexical subs and private methods. I stubbed in conditionals for all of these cases that throw unimplemented exceptions, so people didn't use them expecting that because they parse, they will also work.

So, now there's a bunch of stubs in there for another bunch of OO features and, if nobody beats me to it, I'll be filling some of those out on my next Rakudo day, or maybe before then if time allows (though I'm moving apartment - and country - over the coming week, so I'm not expecting to have much time). A big thanks to Vienna.pm for funding today's work.

Today's work has been a mixture of refactoring and clean-ups that had been on the want list for a while, but just hadn't happened, as well as making some new things work.

First, the initial work I did on types attached them to variables, but what we really needed was a more general way to attach properties. Therefore, there is now a hash of properties instead, where we can stash other stuff.

Next up, I had based pairs on the Parrot Pair PMC, though as Patrick pointed out it's so far off being right for Perl 6 (for example, it's mutable, the Perl 6 one isn't) that we might as well just have our own. Dropping the Parrot Pair PMC and doing that took me all of ten minutes of work, and we get the semantics of pairs a bit more correct too. So that's much cleaner now.

A few days back, dakkar sent in a bug report regarding inheritance. It was almost correct code, but didn't work on Rakudo, since initialization of parent attributes was not yet implemented. I've now implemented this, and I'll borrow the example from the bug report to demonstrate it.

class Foo {
    has $.x;
    method boo { say $.x }
}

class Bar is Foo {
    method set($v) { $.x = $v }
}

my Foo $u .= new(:x(5));
$u.boo;                    # 5

$u= Bar.new(Foo{ :x(12) }); # This is what now works
$u.boo;                    # 12
$u.set(9);
$u.boo;                    # 9


This is not some magical hacky syntax just to make constructors work; you can use it in the general case to associate some vivification data with a proto-object, which gives you a copy of it back with the data attached. It's a bit like currying the object instantiation. So after making the above work, it wasn't much more work to get the following working.

class Foo { has $.x }
my $foo42 = Foo{ :x(42) };
my $test = $foo42.new();
say $test.x; # 42


Note that the original Foo itself isn't changed. We'll have to revisit this again later, because the way I've done it now doesn't have the lazy semantics it's eventually meant to have. It makes the common use case work, though.

With some time spent on objects, I moved onto some improvements to regex stuff. The upshot of this is that you can now use grammar to group regexes into a namespace.

grammar Test {
    regex Load { \d+s };
    rule Loads { <Load> of <Load> };
}
if "100s of 1000s" ~~ Test::Loads { say "yes" }
yes


Note that this is just the start of grammars; inheritance doesn't yet work and you can't smart-match against them yet. It's a stop forward, though.

If you're an avid Rakudo follower, you'll have noted that regex, rule and token all (wrongly) did the same thing before today. I've fixed that too now (there was some behind the scenes work in being able to pass options to the compiler that will be useful elsewhere, as to users of the Parrot Compiler Toolkit in general). In a nutshell, token and rule don't backtrack, where as regex does, and additionally rule translates spaces to the rule, whereas normally they have no effect on the match.

# Demonstrating :ratchet semantics (rule like token here).
regex WillBT { a*a }
token WontBT { a*a }
if "aaa" ~~ WillBT { say "yes" } else { say "no" }
yes
if "aaa" ~~ WontBT { say "yes" } else { say "no" }
no

# Demonstrating :sigspace semantics.
regex Test1 { \d \d };
rule Test2 { \d \d };
if "12" ~~ Test1 { say "yes" } else { say "no" }
yes
if "1 2" ~~ Test1 { say "yes" } else { say "no" }
no
if "12" ~~ Test2 { say "yes" } else { say "no" }
no
if "1 2" ~~ Test2 { say "yes" } else { say "no" }
yes


So, that's what got done today. I'd like to thank Vienna.pm for funding this work, and hope you'll have fun playing with it, breaking it and reporting bugs. :-)

First of all, before I dig into what my recent Rakudo hackings have been, I'd like to thank Vienna.pm for funding me to work on Rakudo. I will be working one full day a week on Rakudo from now on, at least for the next three months and, hopefully, longer. Today is the first day I'm working under this funding, so I'll be posting again later on today about what I got done. This post is just to update you on little bits that I've been doing, but didn't get written up yet.

First of all, you can now use the .= operator.

class Foo { }
my Foo $x .= new();


Here we call the 'new' method on $x, which we know is of type Foo thanks to the type declaration, and assign what it returns - namely, an instance of Foo - to $x. I did initially put this in a while ago, but it was a tad buggy and I wanted to get those worked out before posting it. That's been done, so happy playing. (And note you can use it in places other than declarations too.)

Additionally, some very basic multi-method dispatch based upon types is now in place. You can only use class names, not constraints or role names at the moment for the types, and certainly not more complex types than that. However, it's a start and allows us to run the following example.

class Thing {}
class Rock is Thing {}
class Paper is Thing {}
class Scissors is Thing {}

multi sub defeats(Thing $t1, Thing $t2) { 0 };
multi sub defeats(Paper $t1, Rock $t2) { 1 };
multi sub defeats(Rock $t1, Scissors $t2) { 1 };
multi sub defeats(Scissors $t1, Paper $t2) { 1 };

my $paper = Paper.new;
my $rock = Rock.new;

say defeats($paper, $rock); # 1
say defeats($rock, $paper); # 0


Finally, I put in a small optimization to avoid having to run some runtime type-checks when we can statically determine they're not needed. This should help performance a little.

The Perl 6 design team met by phone on 23 April 2008. Larry, Allison, Patrick, Jerry, Will, Jesse, Nicholas, and chromatic attended.

Jerry:

• we're in a period where everyone's trying to break Parrot
• they're adding new features and accidentally breaking thing
• but they're fixing it
• it's a good part of the cycle
• people fix it
• we don't have a build farm, so we can't test everywhere before committing to trunk

Patrick:

• I thought that was the point of the release cycle

Jerry:

• some people have suggested that we always keep trunk building and passing tests
• but we don't have the means to do that
• especially when we're playing with config
• moving on, the big news is that TPF has six slots in Google's Summer of Code
• one of them is fleshing out the Perl 6 test suite
• we've needed someone to spearhead that
• having a funded new contributor is wonderful
• two Parrot-related projects
• one is generating NCI stubs
• Kevin Tew, a long-time contributor
• the other is the incremental GC specified in the PDD
• that's Andrew Whitworth
• there's also an ASF project for integrating the GC from Apache Harmony into Parrot
• they've wanted to release it as a standalone library
• Parrot's the first test of a standalone system

Nicholas:

• nice that it doesn't count against TPF's slot list

Jesse:

• and it's nice visibility for Parrot from another group

Jerry:

• I finally have six weeks of no plans to travel
• should be able to devote more time to Parrot and Rakudo
• looking forward to that

Larry:

• getting some hacking in on my two pet bugaboos, the longest token matcher and match object generator
• I refactored the matcher
• it still uses TGE
• instead of lumping all of the expect term possible tokens (that is, all of them) into one bucket it separates them into buckets based on their first letter
• it's a one-level tree
• we can build a much smaller DFA for the regexes that start with that letter
• it caches that, of course
• can get an instant reject if the next token can't possibly start with that letter
• also flattened out all of the rules such that the list of tokens is easy to read
• if the first probe with the DFA engine fails, I can take that small set of tokens that start with the same letter, run all of those rules, and sort from longest to shortest
• preserves the token matching order without building huge DFA structures
• as a backoff strategy, that will scale pretty well
• refactored the parameter passing on the matcher side (STD 5)
• instead of passing an initial array of random things, I have parameters
• constructs match objects more correctly
• in the sense that it gets all the information it's supposed to have
• also has some attachments where it shouldn't have
• I'm cleaning that up
• that should scale pretty well

Jerry:

• is there a drop in memory usage?

Larry:

• I haven't measured
• I'm sure that not feeding 800 regexes to TRE at once will make it allocate 17 megabytes on the stack
• it might still be allocating too much for some of the larger things
• I'm still aiming for correct, as opposed to fast
• just trying to bear fast in mind
• the longest token matcher now returns a linked list of states
• not a string
• should be a lot faster; easier to cache
• functionally it's the same as before
• one of those things you don't even have to measure to know it'll be faster
• trying to avoid the bugaboo of premature optimization by doing what I know will be efficient to begin with
• all the while trying to make the thing work
• it has a good chance of being pretty speedy
• my talk in Tokyo will be about all of the places where the current grammar allows extensibility
• it'd be nice to be able to demonstrate some of that

Allison:

• getting work done again
• launched the Strings PDD
• list of tasks for concurrency that I'm breaking down into smaller pieces
• may post what I have now, and leave other people to break them down

Patrick:

• are they parallelizable? :)

Allison:

• many of them are
• there are some bigger things, like switching the exception system over to the event handler
• otherwise, just life stuff

Patrick:

• had paying work come up this past week, so not a lot of actual coding
• need to type the milestones document
• it's all in my head
• managed to remove a lot of unused code thanks to chromatic's post about possible optimizations
• mostly just cleanup
• but helped me figure out things which will feed into my redesign of PGE for longest token matching
• should be able to return to direct Rakduo hacking later this week

Will:

• various Parrot cleanups
• TPF quarterly grant proposals are due at the end of the month
• haven't seen anything come in yet

Allison:

• they're queued in an RT queue
• I don't know if grant members have access to that queue

Will:

• we do
• please, get your proposal in now, sooner than later
• that goes for you on the call as well as people reading the minutes

c:

• mostly spent the past week optimizing Parrot and Rakudo
• looks like it's the building speed is twice as fast as when I started
• runtime is faster too, but the optimization is compilation time
• found some infelicities that need more design thought
• but I'm happy to put these improvements in now and take them out later when the design improves around them
• hope to start adding new features again soon

Patrick:

• most of the test execution time is in compilation
• how useful would it be to compile Rakudo to standalone PIR?

c:

• I'd find it useful
• but I'd find about ten things useful with all I work on
• so not a blocker at the moment

Jesse:

• how far will that get you toward native executables?

Patrick:

• the existing trick for building perl6 would work
• but it's not the same

c:

• if it takes an hour or two, it would help me with debugging and profiling
• if it takes more time, it's not that important

Patrick:

• we have to figure out runtime deployment issues for the Perl 6 runtime library

Will:

• we could add the requirement to run from languages/perl6/ right now

c:

• that's fine by me for now

Patrick:

• that's an afternoon job, not too bad
• what do we need to do to get the Perl 6 and Parrot pages up to date?

Will:

• I'll work on that

Nicholas:

• why is C99 useful to Parrot and the compiler tools?

c:

• front-end parsing for C header files to build NCI declarations automatically
• the backend is pretty easy, that's thunk generation
• the front-end keeps people from having to write boilerplate code by hand
• generate the front-end once, where you have the headers, and then you can run the generated code anywhere even if you don't have the headers

Jesse:

• how does that compare to Python's ctypes?

c:

• as I understand it, they have the nice backend stuff
• not so sure about the front-end
• my P5NCI is nicer, if incomplete
• just haven't had time to work on it...

Jesse:

• if you put in for a TPF grant, that would be very useful

c:

• get me a clone first, and you have a deal

Jerry:

• Allison, we talked about implementing return
• that requires tying in exceptions to the concurrency scheduler?

Allison:

• yes
• just not implemented yet
• when we did exceptions, we didn't have concurrency
• so it's on the top of my list to tie them together

Will:

• Tcl's already using exceptions to handle return, break, and continue

Allison:

• right now, you can't have an exception handler which is a full subroutine

Patrick:

• I'm not sure we need one for that feature
• every subroutine block decorated as such in PAST puts an exception handler in the block
• if any nested block throws a return type exception, it grabs the arguments, does what it has to, and then does a Parrot return

Allison:

• if that's what you need, go ahead and do it
• I thought there are some features you didn't have yet

Patrick:

• I thought there might be some opcodes I needed, like handled
• but we can do something now
• might not be completely optimal
• but it's just packaging things up now
• I have something I think will work
• it's not trivial, but I'm 80% confident
• just a matter of sitting down and doing it

Allison:

• the concurrency stuff will be there before the next release
• might not want to roll it in before the release
• but it'll be there soon

Patrick:

• I want to get return in for the April 15 milestone we're behind on

Jerry:

• have you put in tickets for the breakdown of specific tasks?

Allison:

• I've never done tickets for that
• just sent mail to the list of the tasks
• handed them out to people as they volunteered

c:

• can you put them on a wiki page?
• some of the other committers wanted that

Allison:

• I can do that

The Perl 6 design team met by phone on 16 April 2008. Larry, Allison, Jerry, Will, Nicholas, Jesse, and chromatic attended.

Jerry:

• the release went pretty smoothly
• had some help
• the make release target is broken on Windows, and that's the only platform I had out here
• we'll fix that before the next release from Windows
• talking to Andy Armstrong about getting Test::Harness to 3.0 and subclass TAP::Parser so that we can report Rakudo's fudge tests better
• every fudged test is a failure right now, even if all subtests pass
• I talked Jonathan into implementing simple MMD in Rakudo
• chromatic wrote about it
• but it's broken in the release (and only the release)
• put some work into internationalization
• need to figure out the make rules
• then I can put more work into localization too
• trying to secure the parrot.org domain too
• TPF is likely to get five slots for GSoC
• some will be Parrot and Perl 6 related
• the official announcement is on Monday
• trying to encourage others to take on responsibility
• seems to be working
• some of the committers I've mentored are becoming mentors themselves

Patrick:

• mostly reviewing different things
• working with Jonathan on his various objects and MMD implementations
• will check in my Rakudo milestones document tonight or tomorrow morning
• cleaning up bug reports, closing tickets, catching up on patches
• need to spend a little more time on paying work this week

Will:

• trying to get the most recent release bundled as a macport
• there's apparently a build issue since the previous macport
• should have an easy way to install Rakudo as perl6 after that gets straightened out too
• still trying to cut dead things out of Parrot

c:

• poked at optimizations per Patrick's request
• sped up OO and Rakudo by about 40%
• not as much as I wanted, but you notice it
• looked at a few more optimizations, but they're bigger and take more work
• GC is the biggest, so let's hope we get that as one of the GSoC projects
• think I can get the profiling core to emit Callgrind-compatible output for PIR
• I know mostly how to do it now

Larry:

• spending a lot of time tweaking a new laptop and getting it all set up and customized nicely
• feebly trying to keep up with the onslaught in p6l
• I haven't been keeping up, but I'm keeping my eye out for things going off track badly
• plotting how to get rid of TRE as my longest token matcher
• want something that will scale better
• the handwriting was on the wall the first time I went in it with the debugger
• for the regex matching where a token is expected, TRE was allocating 17 MB on the stack
• I have some ideas for something with better semantics and less memory usage
• TRE optimizes for running one regex a lot over a lot of data, rather than running a lot of regexes

Jesse:

• spent a lot of time in discussions about funding Perl 5 hackers
• I see François Perrad has released a Win32 binary of Parrot

Jerry:

• he's been doing that for the past few releases
• just a release in binary form, not a fork or branch

c:

• what's the status of Parrot in Debian?

Allison:

• one final build bug in IA-64
• the guy with the box should be getting to it this week
• we're planning to put up the 0.6.0 release
• if it doesn't go soon, I'll just put the F<.deb>s on the site
• we might put IA-64 builds on our platform wishlist

c:

• we need someone who knows how to fix them too

Allison:

• it was a PGE bug unrelated to the arch, I think, in 0.4.0
• we just need the bug confirmed fixed on that arch for Debian

Jerry:

• PAUSE kinda stinks
• I hate that we don't know if we'll have an authorized release of Parrot until it gets uploaded
• PARROTRE lacks permissions on eight modules
• all of which have been refactored on something else
• they're all related to the configure or test system

Allison:

• we don't have to distribute through PAUSE

Jerry:

• we should not index those modules anyway

c:

• they're not that useful outside of Parrot anyway

Will:

• the release link on the Parrotcode site, http://www.parrotcode.org/release/devel, links to the CPAN download
• there's a lag between the update and the availability

Jerry:

• we could upload to Parrotcode first
• upload to the CPAN from that site

c:

• I'm not sure why we should index the config::* modules
• the only ones I care about are in Parrot::Embed

The Perl 6 design team met by phone on 09 April 2008. Larry, Allison, Patrick, Will, Jerry, Jesse, Nicholas, and chromatic attended.

Larry:

• trying to support John Dlugosz's clarification work on the spec
• working over the STD 5 grammar
• have it parsing statement lists now
• mainly revising the arcane quoting parsing mechanism it currently has
• plenty of refactoring and cleanup there
• re-evaluating how the tiny microlanguages floating around work
• the regular expression grammar is one sublanguage, a derived language
• there are all these dynamically-scoped minilanguages with terminators
• those tend to change depending on context
• trying to get those terminators to pass up and down the hierarchy efficiently and independently of other sets of tokens, while at the same time meshed in with them is an interesting challenge
• mix them in with the current set of longest tokens against which you're matchin?
• or give your set of terminators a higher precedence?
• an interesting little problem
• mixing them in might make you regenerate your lexer continually depending on your lexical context
• but if you mess with precedence, you might recognize a terminator that's actually a prefix of a longer token
• that could be anti-social
• plays out with all the quoting
• all these little minilanguages depending on which switches you've turned on
• if not terminators, it's at least sets of escape sequences
• also recognizing the terminator at the end of the string
• needs to be a dynamic solution there
• either integrated or not with the longest token solution
• also thinking ahead to the talks I'm giving over the next few months

c:

• did a little library work in Parrot
• trying to write in my journal more about it
• waiting for a list of small implementation tasks for new features
• will work on some polishing for next week's release
• still poking at profiling, but it's a bigger project to do right than people think
• thinking about how to revise our test suite

Allison:

• did some work on the strings PDD
• just have to check that in; I'll try this week
• spent several hours with Jerry on Sunday
• put together a list of milestones after the current set
• these are the key, critical milestones for 1.0
• talked a little bit about Parrot Foundation ideas
• giving a talk about Parrot at LUG Radio Live on Sunday

Jerry:

• working with Patrick on PAST syntax
• Jonathan has done some work on typing with Rakudo
• he's working out a couple of bugs
• he'll start on type-based MMD soon, per my request
• preliminarily we're getting five slots from Google for SoC
• we're hoping for more, because we have 16 good applications
• we'll know more by the end of the week
• preparing for the next release next Tuesday
• should go smoothly

Patrick:

• finished and posted the Mozilla Foundation report to my journal
• made my travel arrangements for the summer
• also speaking at the Texas Open Source Symposium on April 26
• cleaned up some RT tickets, applied some backlogged patches
• catching up on messages received while I was away
• still some Warnocking going on; trying to fix that
• checked in a patch to Rakudo which tells NQP to treat the input files as ASCII, not Unicode
• speeds up parsing by about 28%
• chromatic, can you profile the results and see what pops up?

c:

• I will
• if there's anything at the C level that looks fixable, I'll do it

Patrick:

• had some talks about smoke testing and such
• getting smoke testing of Rakudo is important now
• we can get a web page up with our status so people can see what works
• I'm banging the drums to getting that working, even if I have to do that myself
• doing some cleanups and enhancements to PCT
• closures, for example
• making things more robust
• need to review patches to Rakudo
• their ability to develop outstrips my ability to review
• good problem to have
• it'll take me a little while to catch up
• in the back of my mind, I'm playing with ideas for PGE
• how to integrate longest token match
• if it's possible to emit code that runs on a different backend
• C might be faster
• might run on a different VM
• that's a ways off though
• I want longest token matching first, sometime this summer

Will:

• TPF has grant proposals open now
• any Perl 6 publicizing we can do is a good thing
• regarding the Smolder grant, one of the original deliverables was setting up Smolder for a project which doesn't make sense for Smolder anymore
• someone's working with the project to set up Smolder for Parrot now instead
• no one on the Grants Committee is complaining

c:

• what's the opposite of complaining? I'm doing that

Richard:

• am I surprised you don't know the opposite of "complaining"?

Will:

• I managed to get access to update the Macport of Parrot
• we'll be able to get the current release in there now
• it's two years out of date at the moment
• code-wise, ripping out all of the deprecated features that I can
• also proposed a new opcode syntax, but had no feedback on it

c:

• made sense to me
• I didn't reply because I wanted to look at the opcode you mentioned
• I can't imagine how it could possibly work
• I just haven't proved to myself that it doesn't

Will:

• me too
• probably not tested

Richard:

• Patrick, I encourage you to look at Smolder for what you want with Rakudo

Patrick:

• I did, and I'm thrilled
• the grant behind it is great

Richard:

• putting in effort on acquiring TPF philanthropy
• a large chunk would go toward Perl 6 development
• hopefully soon we can make announcements on the usual channels about success

Jesse:

• CLKAO and I wrote about a thousand lines of code daily each for the past week
• looking for a thermometer about features and milestones per Audrey's Pugs Apocryphon 1
• nice way of giving people a high-level view of how things are going
• she had seven, each corresponding with a major version of Pugs
• I'm looking for an equivalent view for Rakudo

Nicholas:

• where can a symbol be both a terminator and the prefix of a longer symbol?

Larry:

• regular expressions have && and ||, but we have & and | of tighter scope
• if the lower-level rule looks for |, it'll see that in ||
• but it can't see that if there's a || which terminates that expression
• if you have a quoted construct and something that interpolates based on a multi-character token, parsing could hit that
• it's more in terms of regular expressions
• dipping into sub-languages in general
• how do you terminate a block of APL plopped into a chunk of Perl code?
• maybe an unmatched right brace corresponds to something in APL
• you need a way to guarantee policy one way or another in those

Nicholas:

• like the C++ annoyances about >>

Larry:

• you can have an assertion in angle brackets that contains something in angle brackets

Nicholas:

• looks like right shift

Larry:

• especially as double angle brackets by themselves can be their own token

Nicholas:

• will these new grants for TPF occur under the new grants scheme?
• the 2008 Q2 grants?

Richard:

• there's been some discussion about that internally
• it won't happen in full this quarter
• I've suggested that Alberto ask all grant submitters if they're comfortable having their proposals discussed publicly for now

Nicholas:

• I know Andy Armstrong and Merijn worked on parallel testing at Oslo QA
• I know TAP::Harness can run all tests in parallel
• the Perl 5 core needs an API to say that some tests had to run in parallel
• that API is tricky
• they may have solved it
• the Perl 5 core runs into problems with temporary file collisions

c:

• we have the same problem in Parrot
• not necessarily the PIR files written by the Perl 5 test drivers
• but it's solvable

Nicholas:

• Andy Armstrong used DTrace on Mac OS X to find these
• checking file opens and closes

On the train over to Stockholm after the hackathon and on the plane back to Spain a day later, I implemented various cases of 'handles' (not all of them, since the wildcard ones are trickier - they will get done probably along with a whole load of other work on attributes that needs doing). The 'handles' trait verb is the thingy that lets you auto-generate methods that delegate to a methods on an attribute. Here's some examples of what you can do with this:

class Bar { method a { say "a" }; method b { say "b" } }
class Foo1 { has $x handles 'a' } # one method
my $test = Foo1.new(x => Bar.new()); $test.a()
a
class Foo2 { has $x handles <a b> } # several
my $test = Foo2.new(x => Bar.new()); $test.a(); $test.b()
a
b
class Foo3 { has $x handles :mya('a') } # rename one
my $test = Foo3.new(x => Bar.new()); $test.mya();
a
class Foo4 { has $x handles (:mya('a'), :myb('b')) } # rename many
my $test = Foo4.new(x => Bar.new()); $test.mya(); $test.myb();
a
b


So, that's another small piece of the Perl 6 implementation puzzle put into place.

I spent the weekend and most of Monday at the Oslo QA Hackathon. While I'm not a QA expert, it did provide me with time and a relative lack of distractions to get some work done on Rakudo. It was also nice seeing lots of Perl folks, some of whom I've not seen in quite a while. The Hackathon has had a very productive atmosphere with lots getting done; in this post I'll describe some of my Rakudo exploits.

The biggest thing I have been working on is type annotations. This now works with both variable declarations and in the signatures of subroutines and methods, although writing them in a multi-method doesn't lead to any type-based MMD just yet. Attributes of classes can't have types yet either. There are some quirks with various built-in types (including Str and Code types), though many others work fine (Int, Num, Pair, etc). With your own classes, roles and subset types, it should work just fine.

So, for example, if you declare a variable as having type Int, then try and assign a string to it, you'll get a type check failure (we'll get a nicer error message in the end).

my Int $x = 42; # this is ok
my Int $x = "hello"; # this is not
Type check failed


Here's a class example.

class Foo { }
class Bar is Foo { }
class Baz { }
my Foo $x; $x = Foo.new(); # ok, Foo is a Foo
my Foo $x; $x = Bar.new(); # ok, Bar is a Foo
my Foo $x; $x = Baz.new(); # not OK
Type check failed
my Bar $x; $x = Foo.new(); # not OK; Foo is not a Bar
Type check failed


Refinement types also work with this.

subset EvenInt of Int where { $_ % 2 == 0 };
my EvenInt $x = 4; say $x;
4
my EvenInt $x = 4; say $x; $x = 3; say $x;
4
Type check failed


You can do all of this with parameters too, as well as declaring an anonymous refinement.

sub Test(Int where { 0 < $_ <= 100 } $x) { say $x }
Test(50)
50
Test(0) # fails constraint
Parameter type check failed
Test("50") # not an Int
Parameter type check failed


Of course, if we drop the Int from the above, then we get coercion:

sub Test(where { 0 < $_ <= 100 } $x) { say $x }
Test(50)
50
Test("50") # matches constraint when numified
50
Test(0) # still fails it, as expected
Parameter type check failed


Getting this to work forced me to refactor the type hierarchy somewhat - a job that needed doing, but that I'd been putting off (partly out of not trusting myself to do it right). It was a tad nasty, and needed some fixes inside Parrot too. However, the net result is that the majority of built-in object types, such as Int, now inherit from Any (which we didn't have before), and that in turn inherits from Object. The Any distinction will be needed to get junction auto-threading of arguments to work properly; my initial attempt was inefficient and broken. We'll most likely need to do HLL type mapping and similar before that will really work too, though. But anyway, now the following things give what you'd expect.

if 42 ~~ Any { say "yes" }
yes
if 42 ~~ Object { say "yes" }
yes
if 42 | 43 ~~ Any { say "yes" }
if 42 | 43 ~~ Object { say "yes" }
yes
if 42 | 43 ~~ Junction { say "yes" }
yes

I've done a few other things, but I've got a work meeting here tomorrow, so I'm going to sleep now and write about them soon.

The Perl 6 design team met by phone on 02 April 2008. Larry, Allison, Patrick, Jerry, Will, Jesse, Nicholas, and chromatic attended.

Larry:

• things are very busy
• working on a very interesting problem at work that's eating my brain
• sneaked in a few e-mails and synopsis changes
• not much work on the STD parser, just a few nips and tucks
• nothing earth-shaking, mostly just tune-ups in response to John M. Dlugosz and others

Patrick:

• things are going well
• wrote a draft of the second report for the Mozilla Foundation grant
• will post it soon, perhaps tonight
• there has been a ton going on
• this'll be the first in a series of postings to catch up on what's happened the last three months
• closed a few old tickets related to a deprecated feature
• everything that uses PGE is on the new syntax
• reviewed the PCT tutorial and sent kjs some comments and clarifications
• halfway through that
• the changes are all pretty minor
• the tutorial pointed out some places where PCT needs to evolve, and I'm working on that
• getting ready for some heavier Rakudo hacking and reviewing the test suite

Allison:

• finished the main edits to the Strings PDD
• still adding the API reference
• need to ask Simon and Larry to review it
• this is the last PDD in one of the milestones
• working on the next milestone schedule, prioritizing those
• also breaking up tasks for the concurrency milestone

Jerry:

• doing a lot of meta-work, not a lot of work
• four Parrot proposals and one Perl 6 proposal in GSoC
• maybe a few more on the way
• preparing for conference season, especially the Parrot and Perl 6 workshop at YAPC::NA
• hope to catch up with Allison while she's in town
• though we'll probably do more meta work

Will:

• lots of Parrot stuff
• cleaning up build dependencies
• removing some long-deprecated features
• hope to merge back our branch after the next release
• keeping on track with RT
• talked to kjs about shoehorning his tutorial into a compiler tools PDD
• it's a milestone, and one of the last ones
• Allison's looking after the remaining PDDs which aren't part of the milestones

c:

• working with Coke to remove deprecated features
• fixing a few segfaults and crashes, going through open tickets for ideas
• working with a few new contributors on IRC

Jerry:

• they came from Jim Keenan's visit to Toronto
• he's preparing for the workshop at YAPC::NA
• found people with various untested platforms

c:

• that's been very helpful
• adding a few new features
• talked with Mitchell about his plans for a while yesterday

Jesse:

• Patrick and Jerry, what do you think about my request to describe the future of Rakudo in chunkable pieces?

Patrick:

• that's one of my goals
• I'm doing a small part of that in my report to the Mozilla Foundation
• in a longer-term goal, I want to describe what will get us through the summer

Will:

• that's been on my TODO list for Parrot as well
• Google Docs now supports Gantt charts in their spreadsheets
• I've had positive feedback on the RT widget I posted in there
• having something shiny and available might be good for marketing

Jesse:

• even having something internal that's plain text would be good

Will:

• we have existing milestones to look at as a start

Patrick:

• who's planning to go to OSCON?
• will we have Perl 6 meetings before or after OSCON?

Allison:

• the weekend before I have a meeting
• the weekend after might work
• are other people around?

Jesse:

• I'm mostly free

Patrick:

• I was thinking of flying out Saturday afternoon
• might need to find a place to stay

Will:

• I won't be at OSCON
• is anyone going to YAPC::EU this year?

Jesse:

• probably, but not sure

Patrick:

• arranging for funding, would like to go

Allison:

• hadn't planned on it
• but if there's a group of hackers who'd like to hack, I can arrange it

Will:

• there's a chance I might be able to go

Nicholas:

• what's the replacement for IMCC?

Jerry:

• PIRC
• much cleaner
• still lex and yacc

Patrick:

• would the bytecode PMCs be a good Summer of Code project?

Allison:

• it's pretty big
• a C-based bytecode generation library might be a good project

Patrick:

• PCT could use that instead of going to PIR and then through IMCC
• I'll add that to the wiki

Nicholas:

• if the end date is in August, what if it fails and we don't know until then?

Patrick:

• it's not on the critical path

c:

• we want it eventually, but...

Jerry:

• we can go to Parrot 1.0 without PIRC

Allison:

• definitely

Patrick:

• but if someone wants to do it outside of GSoC that's fine too

c:

• I'll convince TPF to give them a medal

Jerry:

• we have five days to convince someone to do this project though
• is the longest token matcher on the critical path to Perl 6.0?
• any estimate on that, Larry?

Larry:

• what I have almost works in prototype
• it works most of the time
• getting it robust enough to where I can use it on the grammar itself is harder to estimate
• the main problem right now is that I'm prototyping it with tagged regular expressions if the longest token actually matches and succeeds
• one of the ways we've defined the matching to work is that you can put assertions after the match which can fail and nullify the match
• so the engine has to have a way of backing off on a match
• TRE isn't up to that in its current form
• I might have to read the code and tweak it
• long-term, we should write something that does longest token matching in Perl 6 which does a parallel NFA traverse of the longest toking rules with the requisite backoff semantics
• I had one of those half-written some months ago before I decided to prototype it with TRE

Jerry:

• what would it take to get concurrency working in Rakudo based on what's in Parrot already?

Larry:

• I meant character matching in parallel, not OS-level concurrency

Patrick:

• more important is MMD

Allison:

• that's the next one

Patrick:

• a lot of things in Perl 6 depend on MMD working
• or solving problems for you
• otherwise you have to emulate it to make it look like what's happening
• that's where Jonathan's holding up on stuff
• Rakudo will too
• MMD, strings, and concurrency are my priorities
• may not be true from the IO perspective, though
• my working plan for Rakudo is that I want to get it along the way with features
• then step back and revisit PGE and the regex engine, starting with Larry's prototype
• I'd like to do that before the summer conferences to sit down with Larry face-to-face
• may or may not come to pass
• but my working plan is to look at longest-token matching in PGE in mid-to-late summer

Allison:

• what's the blocker with Rakudo and MMD?

Patrick:

• Rakudo doesn't implement it yet
• the next blocker is that we'll use MMD this month
• then I'll be able to identify problems and potential problems

Allison:

• those notes'll feed in to the real, final implementation