Recently in Perl 6 Category

I started out the day by looking through the RT queue for Rakudo. Two tickets were already dealt with, so I just closed those. Another was a bug report concerning assigning undef to typed variables. Doing:

my Int $x = undef;


Would give a type check failure. This is now resolved. Furthermore, if your type is a class name, then assigning undef at any point to it will result in it holding the protoobject for that class again. I also took a moment to post to Perl 6 Language to get some clarifications on what "my TypeName $x;" left $x being when $x was a role, subset type or a junction of types.

Last week I started getting grammars in place. I got so far as having the regex live in the correct namespaces, but that didn't make grammars at all class-like, which is how they should be. This week I set out to fix that. Grammars now get protoobjects too, which you can call .WHAT and so forth on. Furthermore, I got inheritance working and also smart-matching against a grammar, which runs it's TOP rule. Therefore you can now run the following example in Rakudo.

grammar Loads { regex Lots { \d+s } };
grammar Many is Loads { rule TOP { of } };
if "100s of 1000s" ~~ Many {
    say $/; # 100s of 1000s
}


Having closed four RT tickets so far, I took a look through there to see what else there was. There was one that did most of what was needed to implement the .perl method on Junctions (which you can call, in theory anyway, on anything to get a Perl representation of it). I did the required fixes and applied it, but realized in the process that we were missing .perl for any of the really fundamental types, so I added it for Num, Int and Str.

With that done, I spent a little time on the S12 tests. I added fudge directives to get one of the that failed to parse to do so, and added an extra test. I plan to add much more here and flesh out the tests quite a bit over time.

Turning back to the OO support, I did some updates to the grammar that both brought us closer to STD.pm - the official grammar - and added the ability to parse a range of extra things. The first grammar changes were related to method calling. Normally you call a method just with ".", but private methods are called with "!". Additionally, there are ways to call sets of methods with quantifiers (.?, .+ and .*, with meanings analogous to those in regexes). Finally, I added the ability to scope declarations of routines too, so we can now parse lexical subs and private methods. I stubbed in conditionals for all of these cases that throw unimplemented exceptions, so people didn't use them expecting that because they parse, they will also work.

So, now there's a bunch of stubs in there for another bunch of OO features and, if nobody beats me to it, I'll be filling some of those out on my next Rakudo day, or maybe before then if time allows (though I'm moving apartment - and country - over the coming week, so I'm not expecting to have much time). A big thanks to Vienna.pm for funding today's work.

Today's work has been a mixture of refactoring and clean-ups that had been on the want list for a while, but just hadn't happened, as well as making some new things work.

First, the initial work I did on types attached them to variables, but what we really needed was a more general way to attach properties. Therefore, there is now a hash of properties instead, where we can stash other stuff.

Next up, I had based pairs on the Parrot Pair PMC, though as Patrick pointed out it's so far off being right for Perl 6 (for example, it's mutable, the Perl 6 one isn't) that we might as well just have our own. Dropping the Parrot Pair PMC and doing that took me all of ten minutes of work, and we get the semantics of pairs a bit more correct too. So that's much cleaner now.

A few days back, dakkar sent in a bug report regarding inheritance. It was almost correct code, but didn't work on Rakudo, since initialization of parent attributes was not yet implemented. I've now implemented this, and I'll borrow the example from the bug report to demonstrate it.

class Foo {
    has $.x;
    method boo { say $.x }
}

class Bar is Foo {
    method set($v) { $.x = $v }
}

my Foo $u .= new(:x(5));
$u.boo;                    # 5

$u= Bar.new(Foo{ :x(12) }); # This is what now works
$u.boo;                    # 12
$u.set(9);
$u.boo;                    # 9


This is not some magical hacky syntax just to make constructors work; you can use it in the general case to associate some vivification data with a proto-object, which gives you a copy of it back with the data attached. It's a bit like currying the object instantiation. So after making the above work, it wasn't much more work to get the following working.

class Foo { has $.x }
my $foo42 = Foo{ :x(42) };
my $test = $foo42.new();
say $test.x; # 42


Note that the original Foo itself isn't changed. We'll have to revisit this again later, because the way I've done it now doesn't have the lazy semantics it's eventually meant to have. It makes the common use case work, though.

With some time spent on objects, I moved onto some improvements to regex stuff. The upshot of this is that you can now use grammar to group regexes into a namespace.

grammar Test {
    regex Load { \d+s };
    rule Loads { <Load> of <Load> };
}
if "100s of 1000s" ~~ Test::Loads { say "yes" }
yes


Note that this is just the start of grammars; inheritance doesn't yet work and you can't smart-match against them yet. It's a stop forward, though.

If you're an avid Rakudo follower, you'll have noted that regex, rule and token all (wrongly) did the same thing before today. I've fixed that too now (there was some behind the scenes work in being able to pass options to the compiler that will be useful elsewhere, as to users of the Parrot Compiler Toolkit in general). In a nutshell, token and rule don't backtrack, where as regex does, and additionally rule translates spaces to the rule, whereas normally they have no effect on the match.

# Demonstrating :ratchet semantics (rule like token here).
regex WillBT { a*a }
token WontBT { a*a }
if "aaa" ~~ WillBT { say "yes" } else { say "no" }
yes
if "aaa" ~~ WontBT { say "yes" } else { say "no" }
no

# Demonstrating :sigspace semantics.
regex Test1 { \d \d };
rule Test2 { \d \d };
if "12" ~~ Test1 { say "yes" } else { say "no" }
yes
if "1 2" ~~ Test1 { say "yes" } else { say "no" }
no
if "12" ~~ Test2 { say "yes" } else { say "no" }
no
if "1 2" ~~ Test2 { say "yes" } else { say "no" }
yes


So, that's what got done today. I'd like to thank Vienna.pm for funding this work, and hope you'll have fun playing with it, breaking it and reporting bugs. :-)

The Perl 6 design team met by phone on 23 April 2008. Larry, Allison, Patrick, Jerry, Will, Jesse, Nicholas, and chromatic attended.

Jerry:

• we're in a period where everyone's trying to break Parrot
• they're adding new features and accidentally breaking thing
• but they're fixing it
• it's a good part of the cycle
• people fix it
• we don't have a build farm, so we can't test everywhere before committing to trunk

Patrick:

• I thought that was the point of the release cycle

Jerry:

• some people have suggested that we always keep trunk building and passing tests
• but we don't have the means to do that
• especially when we're playing with config
• moving on, the big news is that TPF has six slots in Google's Summer of Code
• one of them is fleshing out the Perl 6 test suite
• we've needed someone to spearhead that
• having a funded new contributor is wonderful
• two Parrot-related projects
• one is generating NCI stubs
• Kevin Tew, a long-time contributor
• the other is the incremental GC specified in the PDD
• that's Andrew Whitworth
• there's also an ASF project for integrating the GC from Apache Harmony into Parrot
• they've wanted to release it as a standalone library
• Parrot's the first test of a standalone system

Nicholas:

• nice that it doesn't count against TPF's slot list

Jesse:

• and it's nice visibility for Parrot from another group

Jerry:

• I finally have six weeks of no plans to travel
• should be able to devote more time to Parrot and Rakudo
• looking forward to that

Larry:

• getting some hacking in on my two pet bugaboos, the longest token matcher and match object generator
• I refactored the matcher
• it still uses TGE
• instead of lumping all of the expect term possible tokens (that is, all of them) into one bucket it separates them into buckets based on their first letter
• it's a one-level tree
• we can build a much smaller DFA for the regexes that start with that letter
• it caches that, of course
• can get an instant reject if the next token can't possibly start with that letter
• also flattened out all of the rules such that the list of tokens is easy to read
• if the first probe with the DFA engine fails, I can take that small set of tokens that start with the same letter, run all of those rules, and sort from longest to shortest
• preserves the token matching order without building huge DFA structures
• as a backoff strategy, that will scale pretty well
• refactored the parameter passing on the matcher side (STD 5)
• instead of passing an initial array of random things, I have parameters
• constructs match objects more correctly
• in the sense that it gets all the information it's supposed to have
• also has some attachments where it shouldn't have
• I'm cleaning that up
• that should scale pretty well

Jerry:

• is there a drop in memory usage?

Larry:

• I haven't measured
• I'm sure that not feeding 800 regexes to TRE at once will make it allocate 17 megabytes on the stack
• it might still be allocating too much for some of the larger things
• I'm still aiming for correct, as opposed to fast
• just trying to bear fast in mind
• the longest token matcher now returns a linked list of states
• not a string
• should be a lot faster; easier to cache
• functionally it's the same as before
• one of those things you don't even have to measure to know it'll be faster
• trying to avoid the bugaboo of premature optimization by doing what I know will be efficient to begin with
• all the while trying to make the thing work
• it has a good chance of being pretty speedy
• my talk in Tokyo will be about all of the places where the current grammar allows extensibility
• it'd be nice to be able to demonstrate some of that

Allison:

• getting work done again
• launched the Strings PDD
• list of tasks for concurrency that I'm breaking down into smaller pieces
• may post what I have now, and leave other people to break them down

Patrick:

• are they parallelizable? :)

Allison:

• many of them are
• there are some bigger things, like switching the exception system over to the event handler
• otherwise, just life stuff

Patrick:

• had paying work come up this past week, so not a lot of actual coding
• need to type the milestones document
• it's all in my head
• managed to remove a lot of unused code thanks to chromatic's post about possible optimizations
• mostly just cleanup
• but helped me figure out things which will feed into my redesign of PGE for longest token matching
• should be able to return to direct Rakduo hacking later this week

Will:

• various Parrot cleanups
• TPF quarterly grant proposals are due at the end of the month
• haven't seen anything come in yet

Allison:

• they're queued in an RT queue
• I don't know if grant members have access to that queue

Will:

• we do
• please, get your proposal in now, sooner than later
• that goes for you on the call as well as people reading the minutes

c:

• mostly spent the past week optimizing Parrot and Rakudo
• looks like it's the building speed is twice as fast as when I started
• runtime is faster too, but the optimization is compilation time
• found some infelicities that need more design thought
• but I'm happy to put these improvements in now and take them out later when the design improves around them
• hope to start adding new features again soon

Patrick:

• most of the test execution time is in compilation
• how useful would it be to compile Rakudo to standalone PIR?

c:

• I'd find it useful
• but I'd find about ten things useful with all I work on
• so not a blocker at the moment

Jesse:

• how far will that get you toward native executables?

Patrick:

• the existing trick for building perl6 would work
• but it's not the same

c:

• if it takes an hour or two, it would help me with debugging and profiling
• if it takes more time, it's not that important

Patrick:

• we have to figure out runtime deployment issues for the Perl 6 runtime library

Will:

• we could add the requirement to run from languages/perl6/ right now

c:

• that's fine by me for now

Patrick:

• that's an afternoon job, not too bad
• what do we need to do to get the Perl 6 and Parrot pages up to date?

Will:

• I'll work on that

Nicholas:

• why is C99 useful to Parrot and the compiler tools?

c:

• front-end parsing for C header files to build NCI declarations automatically
• the backend is pretty easy, that's thunk generation
• the front-end keeps people from having to write boilerplate code by hand
• generate the front-end once, where you have the headers, and then you can run the generated code anywhere even if you don't have the headers

Jesse:

• how does that compare to Python's ctypes?

c:

• as I understand it, they have the nice backend stuff
• not so sure about the front-end
• my P5NCI is nicer, if incomplete
• just haven't had time to work on it...

Jesse:

• if you put in for a TPF grant, that would be very useful

c:

• get me a clone first, and you have a deal

Jerry:

• Allison, we talked about implementing return
• that requires tying in exceptions to the concurrency scheduler?

Allison:

• yes
• just not implemented yet
• when we did exceptions, we didn't have concurrency
• so it's on the top of my list to tie them together

Will:

• Tcl's already using exceptions to handle return, break, and continue

Allison:

• right now, you can't have an exception handler which is a full subroutine

Patrick:

• I'm not sure we need one for that feature
• every subroutine block decorated as such in PAST puts an exception handler in the block
• if any nested block throws a return type exception, it grabs the arguments, does what it has to, and then does a Parrot return

Allison:

• if that's what you need, go ahead and do it
• I thought there are some features you didn't have yet

Patrick:

• I thought there might be some opcodes I needed, like handled
• but we can do something now
• might not be completely optimal
• but it's just packaging things up now
• I have something I think will work
• it's not trivial, but I'm 80% confident
• just a matter of sitting down and doing it

Allison:

• the concurrency stuff will be there before the next release
• might not want to roll it in before the release
• but it'll be there soon

Patrick:

• I want to get return in for the April 15 milestone we're behind on

Jerry:

• have you put in tickets for the breakdown of specific tasks?

Allison:

• I've never done tickets for that
• just sent mail to the list of the tasks
• handed them out to people as they volunteered

c:

• can you put them on a wiki page?
• some of the other committers wanted that

Allison:

• I can do that

The Perl 6 design team met by phone on 16 April 2008. Larry, Allison, Jerry, Will, Nicholas, Jesse, and chromatic attended.

Jerry:

• the release went pretty smoothly
• had some help
• the make release target is broken on Windows, and that's the only platform I had out here
• we'll fix that before the next release from Windows
• talking to Andy Armstrong about getting Test::Harness to 3.0 and subclass TAP::Parser so that we can report Rakudo's fudge tests better
• every fudged test is a failure right now, even if all subtests pass
• I talked Jonathan into implementing simple MMD in Rakudo
• chromatic wrote about it
• but it's broken in the release (and only the release)
• put some work into internationalization
• need to figure out the make rules
• then I can put more work into localization too
• trying to secure the parrot.org domain too
• TPF is likely to get five slots for GSoC
• some will be Parrot and Perl 6 related
• the official announcement is on Monday
• trying to encourage others to take on responsibility
• seems to be working
• some of the committers I've mentored are becoming mentors themselves

Patrick:

• mostly reviewing different things
• working with Jonathan on his various objects and MMD implementations
• will check in my Rakudo milestones document tonight or tomorrow morning
• cleaning up bug reports, closing tickets, catching up on patches
• need to spend a little more time on paying work this week

Will:

• trying to get the most recent release bundled as a macport
• there's apparently a build issue since the previous macport
• should have an easy way to install Rakudo as perl6 after that gets straightened out too
• still trying to cut dead things out of Parrot

c:

• poked at optimizations per Patrick's request
• sped up OO and Rakudo by about 40%
• not as much as I wanted, but you notice it
• looked at a few more optimizations, but they're bigger and take more work
• GC is the biggest, so let's hope we get that as one of the GSoC projects
• think I can get the profiling core to emit Callgrind-compatible output for PIR
• I know mostly how to do it now

Larry:

• spending a lot of time tweaking a new laptop and getting it all set up and customized nicely
• feebly trying to keep up with the onslaught in p6l
• I haven't been keeping up, but I'm keeping my eye out for things going off track badly
• plotting how to get rid of TRE as my longest token matcher
• want something that will scale better
• the handwriting was on the wall the first time I went in it with the debugger
• for the regex matching where a token is expected, TRE was allocating 17 MB on the stack
• I have some ideas for something with better semantics and less memory usage
• TRE optimizes for running one regex a lot over a lot of data, rather than running a lot of regexes

Jesse:

• spent a lot of time in discussions about funding Perl 5 hackers
• I see François Perrad has released a Win32 binary of Parrot

Jerry:

• he's been doing that for the past few releases
• just a release in binary form, not a fork or branch

c:

• what's the status of Parrot in Debian?

Allison:

• one final build bug in IA-64
• the guy with the box should be getting to it this week
• we're planning to put up the 0.6.0 release
• if it doesn't go soon, I'll just put the F<.deb>s on the site
• we might put IA-64 builds on our platform wishlist

c:

• we need someone who knows how to fix them too

Allison:

• it was a PGE bug unrelated to the arch, I think, in 0.4.0
• we just need the bug confirmed fixed on that arch for Debian

Jerry:

• PAUSE kinda stinks
• I hate that we don't know if we'll have an authorized release of Parrot until it gets uploaded
• PARROTRE lacks permissions on eight modules
• all of which have been refactored on something else
• they're all related to the configure or test system

Allison:

• we don't have to distribute through PAUSE

Jerry:

• we should not index those modules anyway

c:

• they're not that useful outside of Parrot anyway

Will:

• the release link on the Parrotcode site, http://www.parrotcode.org/release/devel, links to the CPAN download
• there's a lag between the update and the availability

Jerry:

• we could upload to Parrotcode first
• upload to the CPAN from that site

c:

• I'm not sure why we should index the config::* modules
• the only ones I care about are in Parrot::Embed

On the train over to Stockholm after the hackathon and on the plane back to Spain a day later, I implemented various cases of 'handles' (not all of them, since the wildcard ones are trickier - they will get done probably along with a whole load of other work on attributes that needs doing). The 'handles' trait verb is the thingy that lets you auto-generate methods that delegate to a methods on an attribute. Here's some examples of what you can do with this:

class Bar { method a { say "a" }; method b { say "b" } }
class Foo1 { has $x handles 'a' } # one method
my $test = Foo1.new(x => Bar.new()); $test.a()
a
class Foo2 { has $x handles <a b> } # several
my $test = Foo2.new(x => Bar.new()); $test.a(); $test.b()
a
b
class Foo3 { has $x handles :mya('a') } # rename one
my $test = Foo3.new(x => Bar.new()); $test.mya();
a
class Foo4 { has $x handles (:mya('a'), :myb('b')) } # rename many
my $test = Foo4.new(x => Bar.new()); $test.mya(); $test.myb();
a
b


So, that's another small piece of the Perl 6 implementation puzzle put into place.

First off, sorry it's been a while since I last posted. It's mostly been that I've just not done a lot between the Ukrainian and Dutch Perl Workshops and now, which was thanks to being busy with $REAL_LIFE and lacking either time or brain cycles for hacking Rakudo. Thankfully, I'm back into the swing of things now, and it's time to catch up on the blogging.

I'd like to give a belated thanks to everyone at the UPW and DPW for such a good time. I greatly enjoyed attending, meeting people and speaking at both workshops, and Kiev was a more beautiful city than I had imagined it being. The cathedrals and churches there are incredible - I'll surely be coming back to Ukraine to explore some more!

At UPW, some deficiencies in smart matching of arrays was discovered in Rakudo and I promised a fix. I also said I'd write a note about it here "soon", which I failed to do - sorry. I thought I had fixed array comparrison with smart match, but that appears not to be working right now. :-( Also, the Perl 5 and Perl 6 way of checking if a list contains an element are different, and the Perl 6 way was dependent on something else that wasn't implemented (though I've started work on it now, so hopefully we have that soon too). Anyway, it's on the horizon.

I'm hazy on exactly what I got done at DPW and around there, but I know I started on pairs. Thanks to some extra contributions from cognominal++, quite a lot of this is now implemented. As specifying named parameters in a sub was already done, happily it means you can now do things like:

sub sayit (:$what) { say $what; }
sayit(what => "Oo minya pivo!");
Oo minya pivo!

It also works with object instantiation too.

class Foo { has $.a }
my $x = Foo.new(a => 42); say $x.a;
42

As well as the fat arrow syntax, there's also the colon pair syntax, in various forms (we ain't got 'em all yet, but the most useful ones are there).

sayit(:what);
1
sayit(:!what);
0
sayit(:what('privyet'));
privyet
my $what = 'nyam'; sayit(:$what);
nyam

Another thing I've hacked in some basic support for is the subset keyword, which allows you to write refinement types. You can smart-match against the type name to test if the variable meets that type.

subset Guess where { $_ >= 1 && $_ <= 100 };
if 42 ~~ Guess { say "yes" } else { say "no" }
yes
if 142 ~~ Guess { say "yes" } else { say "no" }
no
if 97 ~~ Guess { say "yes" } else { say "no" }
yes
subset LowGuess of Guess where { $_ <= 50; };
if 42 ~~ LowGuess { say "yes" } else { say "no" }
yes
if 97 ~~ LowGuess { say "yes" } else { say "no" }
no
if 0 ~~ LowGuess { say "yes" } else { say "no" }
no

Hopefully more types related stuff coming soon, amongst various other things. I'll be at the Oslo QA Hackathon, and hope to get plenty of free time there, as well as generally while traveling over the coming weekends.

The Perl 6 design team met by phone on 30 January 2008. Larry, Allison, Patrick, Will, Jerry, Jesse, Nicholas, and chromatic attended.

Larry:

• double-booked at work, so I'm kinda busy there
• still doing Perl 6 clarifications as requested
• starting firestorms that I've subsequently ignored on p6l
• some of the $work I'm doing is in the same mindspace as the longest token matcher
• might bootstrap me back into that

c:

• still working on the PMC PDD branch with Allison
• have a couple of bugs to fix there
• a couple of cleanups
• one minor optimization that will reduce (generated) code size somewhat
• another similar optimization to look at, should be more valuable

Patrick:

• unexpected family and house issues came up
• I'm back now
• the plan is to continue on last week's work
• mostly conversions to Rakudo to make it fit what Parrot expects (namspaces and the like)

Jerry:

• made some more spec tests pass with Rakudo
• added type constraint parsing
• thinking about a lot of things
• list context, return exceptions, Perl 6 docs
• very close to committing perl6doc executable for Rakudo

Jesse:

• according to Damian's spec?

Jerry:

• according to Larry's grammar to POD6
• it's small
• it's a start for people who want to hack on the POD6 grammar
• had a discussion with Alias with respect to deterministic parsing
• there's a small patch to prevent Rakudo from running BEGIN blocks
• not sure it's worth applying

Larry:

• see my latest diatribe on p6c

Patrick:

• grammar changes aren't implemented with BEGIN blocks

Larry:

• I don't care if you add the switch

Jerry:

• is it worth adding something to prevent BEGIN blocks from running

Larry:

• I don't see its value, but it shouldn't hurt anything

Patrick:

• just derive a new grammar that disables BEGIN blocks and use that
• shouldn't have to do anything in the compiler to make that happen

Allison:

• spent some time on Mac OS X compilation problems
• seem to have a solution for that
• still working on the PMC implementation
• giving a talk at Linux.conf.au on the Pynie implementation
• it's interesting poking in the guts of NQP and PCT and comparing that to the old PGE/TGE approach

Will:

• my RT viewer is pretty much done
• just need to get an RT instance to play with extracting data in a timely fashion
• working on the Mac port for Parrot, hope to get 0.5.2 in MacPorts in a couple of weeks
• should be simple to keep it updated from then on

Nicholas:

• Rafael wants a lot more test cases for switch in 5.10.1

Patrick:

• we're looking for hints about how to do list context parsing in STD.pm

Larry:

• particle and I talked about that the other day

Patrick:

• that's the next big thing we're missing
• I don't want to hack something together if there's something better in STD.pm

Jesse:

• any blocking?

c:

• I really would love more platform coverage
• a few patches lately that need attention on something more than 32-bit Linux with GCC
• any version of OpenSolaris, just about any BSD would be good, Mac OS X, Windows

Patrick:

• we can set up VMs
• ask people to create their own VM images

c:

• that would work
• ask for freely-redistributable VM images
• and then get non-redistributable OSes to submit to the smoke farm

Patrick:

• I can post that in my journal
• or post a request on Perlbuzz

Latest design meeting minutes are up. I wish there was a summary in there of key interestingness, and what came out of the meeting. I'm glad to see the notes, but there's no analysis for the outsiders to really know what happened, what the meaning is.

If Perl 6 is a little daunting, take a smaller bite by reading one of Adriano Ferreira's Perl 6 microarticles. The index of articles is on the official Perl 6 wiki on the page called Perl 6 microarticles. As of now the list of articles is:

• Introduction to the articles
• Zip
• Stitching
• Repeat operators
• Coercion operators
• Comparisons part I
• Comparisons part II
• Boolean operators
• the Default operator
• Range operators
• Negated operators
• Conditional operator
• Cross operator
• Iterator operator
• Reduce operators
• Reduce operators II
• Mutating operators
• The pair constructor
• Junction operators
• Filetests