Perl 6: June 2008 Archives

[Rakudo spec regression status: 75 files, 1080 passing tests]

Rakudo Perl is now passing over 1000 tests in the official test suite!

For those who are interested, the history of test results is available in CSV format from http://svn.perl.org/parrot/trunk/languages/perl6/docs/spectest-progress.csv .

Thanks to Jonathan, Moritz, Auzon, bacek, particle, Coke, chromatic, and everyone else who is helping to make this happen!

Rakudo day came around again, and I pushed everything else that is keeping me busy these days aside and focused on getting more of Perl 6 implemented. This week there are a handful of new features; as is the norm of late, I have been continuing my focus on the object model and the type system.

I started off the day with some refactoring. I wanted to bring us more in line with STD.pm - the standard Perl 6 grammar - as well as parse some additional bits of syntax that we didn't parse before for some new things that are coming in the not too distant future. Getting us closer to STD.pm's grammar rules could have been done without that much shuffling around, but there was an action method that was more than long enough already lying behind it all. I didn't like the thought of making it even longer and harder to follow when I came to add to it in a week or two, and the re-arrangement of the grammar presented an obvious way to break the action method up into several methods. So, I did that and that bit of the codebase is now a bit cleaner, and ready for when I and others need to build more on top of it.

So, what's new? Probably the most major new feature this week is that you can now compose roles into an object at runtime, mix-in style. Here is an example from the tests to illustrate some of the semantics.

role R { method test { say 42 } }
class C { has $.x }
my $y = C.new(x => 100);
$y does R;
$y.test; # 42 (called method from role)
say $y.x; # 100 - we didn't destroy old attribute values

There is also some special syntax, implemented too, for initializing values of a role that have a single attribute. For example, imagine we wrote an Answer role:

role Answer { has $.answer is rw }


Then we can do the following:

role Answer { has $.answer is rw }
my $x = 100;
$x does Answer(42);
say $x; # 100
say $x.answer; # 42


If you're thinking now, "is this the return 0 but True thingy I once saw", the the answer is almost, yes. However, unlike "does", the "but" operator creates, operates on and returns a copy of the value rather than the original. It also does some special handling of values from enumerations. It is defined in terms of "does", so we're part of the way to "but". Getting the enumerations right is the harder part to implement.

The other major new feature I got in today was generic subs/methods. With these, you capture the type of a parameter in the call signature. You can then use it inside the sub or method. This means that we can write a sub that prints the type of the parameter it is passed just by doing:

sub saytype(::T $x) { say T }
saytype(42); # Int
saytype("Mam pivo"); # Str


You can use T anywhere you would use a type otherwise.

sub test(::T $x) { my T $y = "OH HAI" }
test(42) # Type check failed
test("hi") # Works


Note the variable $y inside the sub is constrained to be the same type as whatever the parameter to the sub was. In the first case, we pass an Int, so $y is constrained to be an Int and the assignment fails. In the second case, we pass a Str, so the assignment is successful. One remaining to-do feature is to let you use T elsewhere in the signature after you have declared it; that will be on my hit list for next week.

Finally, to round off the day, I took something a little easier. I've been eying enumerations for implementation, and the anonymous enum constructor was somewhat easier than the full-blown named one (which I'm still fully getting my head around). Basically, it can take a list of values and hand back a hash mapping each of them to ascending integers.

my %ooks = enum < ook! ook. ook? >;
say %ooks<ook!>; # 0
say %ooks<ook.>; # 1
say %ooks<ook?>; # 2


You can instead write a pair (in a list constructor that parses them as pairs, mind) to specify the starting value, which may be a string too.

my %meta = enum [ :foo('A'), 'bar', 'baz' ];
say %meta<foo>; # A
say %meta<bar>; # B
say %meta<baz>; # C


You can stick a pair at any point in there to change the indexing scheme too, or just use a load of mappings to set up your enum. At this point, it may seem like a fancy way of constructing a hash - and in a sense, it kinda is. However, the non-anonymous case introduces something somewhat more powerful, which as mentioned earlier can be used with "but", and also as a predicate function and in a smart match. Hopefully I get those in sometime in the coming weeks.

So, that's what got done on this week's Rakudo day. Another one next week, and thanks as always go to Vienna.pm for making this possible.

[Rakudo spec regression status: 64 files, 779 tests]

In my previous posts I reported on Rakudo's progress in passing the test suite. For those I had been using the output of Test::Harness to estimate the number of passing tests, but ultimately decided that it wasn't really giving me the information I want in the form I want it. So I wrote a custom test summarizer for Rakudo to make it easier to measure our test passing rate in the spectest suite.

Here's how things have been progressing over the last three weeks:

          Rakudo spectest regression daily results
                          files test pass fail todo skip
2008-05-22 00:00             31  564  223    0    0  341
2008-05-23 00:00             32  569  228    0    0  341
2008-05-24 00:00             32  569  228    0    0  341
2008-05-25 00:00             39  666  310    0    0  356
2008-05-26 00:00             39  666  310    0    0  356
2008-05-27 00:00             39  666  310    0    0  356
2008-05-28 00:00             39  666  317    0    0  349
2008-05-29 00:00             43  774  394    4   15  361
2008-05-30 00:00             43  775  415    0   15  345
2008-05-31 00:00             43  775  415    0   15  345
2008-06-01 00:00             52  892  518    0   15  359
2008-06-02 00:00             55 1012  623    0   15  374
2008-06-03 00:00             55 1012  623    0   15  374
2008-06-04 00:00             55 1012  624    0   14  374
2008-06-05 00:00             58 1107  668    0   15  424
2008-06-06 00:00             58 1110  674    0   14  422
2008-06-07 00:00             59 1139  682    0   16  441
2008-06-08 00:00             59 1139  697    0   17  425
2008-06-09 00:00             59 1139  699    0   15  425
2008-06-10 00:00             59 1139  699    0   15  425
2008-06-11 00:00             59 1145  705    0   15  425
2008-06-12 00:00             59 1145  705    0   15  425
2008-06-13 00:00             60 1148  707    0   15  426
2008-06-14 00:00             60 1148  711    0   15  422
2008-06-15 00:00             63 1201  754    0   15  432
2008-06-16 00:00             64 1226  779    0   15  432

The 00:00 in the above table represents midnight U.S. Central Time. The 'test' column indicates the number of tests run, and the 'pass' column shows how many tests were passing (excluding 'todo' and 'skip'). So, as of June 16 Rakudo is passing 779 tests in its spectest_regression suite.

In order to get a feel for our overall trend, I like to look at week-over-week progress instead of just the daily numbers. So, in the week from June 9 to June 16 we added 779 - 699 = 80 new passing tests. We'll see how things go in the future -- it will all depend on a combination of how quickly spectests can be reviewed and features added to Rakudo.

At the top of my future posts I plan to include a one-line summary of Rakudo's passing rate so that people can continually monitor our ongoing progress without having to scan the article (see the top of this post for an example).

Pm

After taking last week off for workshops, this week I got back to having my weekly Rakudo day. There were various things that were just about ready, and I've spent much of today getting those in place. The result is some new features, some fixes and some work on getting the semantics correct in places that we didn't before.

One of the big changes is that the range operator, "..", will now create a Range object, which is a lazy iterator. I'd done some of the initial work in writing the Range class before now, but using it blocked on some other list refactoring that was needed. That has now been done (thanks to Patrick for this), and today I was able to switch ".." over to creating the Range object rather than just a flat list. It still will be eager in a lot of places, because we haven't got a full lazy lists implementation just yet. However:

for 1..1000000 -> $x {
    say $x;
}


Will now not create an array of a million elements. In fact, it will run in constant space. I also implemented smart-matching against Range objects. This means that you can do things like:

if $x ~~ 1..100 { say "valid percentage" }

Again, this will not create a list 100 elements long, but instead just one Range object that knows its endpoints. There are still various things to do with ranges, including the operators for constructing ranges that are exclusive of their endpoints (and Range objects need some work for this themselves too). Also, we need to implement the :by adverb, but we can't parse adverbs on operators just yet. Oh, and then there's the fun of infinite ranges, but I'm hoping they will just fall out naturally with little to no changes to Range once the Inf type is in place.

I've been mumbling for a while about a bunch of work under the title of mutables. It has very much been an under-the-hood thing that was needed to make other things possible. Today I got us fully switched over to this model for scalars with no additional tests failing. And I also started taking of advantage of it to get a few more things in place.

Before, parameters were passed and were modifiable. However, unless you used "is rw" this should not have been so; they are meant to be readonly by default. Now we have the correct semantics, and "is rw", "is copy" and "is readonly" work too (but writing "is readonly" is a waste of time here, since it's the default anyway; I haven't bothered in the example below).

sub foo($x) { $x = 42; say $x; }
my $a = 100; foo($a); say $a; # Cannot assign to readonly variable
sub foo($x is rw) { $x = 42; say $x; }
my $a = 100; foo($a); say $a; # 42\n42\n
sub foo($x is copy) { $x = 42; say $x; }
my $a = 100; foo($a); say $a; # 42\n100\n


I also got the VAR($x) and $x.VAR macros in place (they are actually compiler macros, not sub or method calls), which let you get at the underlying implementation type - in this case the default Scalar object. In the end, this will let you work with traits and tied containers, though we are some way off being able to implement those yet (quite a few dependencies). For now you can do things like $x.VAR.readonly to find out if a variable is readonly, for example.

I've also started doing some preparations for the Next Big Thing I intend to do some work on during my Rakudo day next week: roles. We already have some basic composition support. I fixed using roles as type constraints on a variable. However, what we have so far isn't even scratching the surface of what we need to fully do roles. Attributes surely aren't done right, for example. I've spent some time reading the appropriate bits of S12, trying to get into my head what it means. I also got to discuss some bits of it with Larry (especially the type parameterization bits), and it seems I'm along the right lines. So, some work coming on roles next week, all being well.

I also fixed a range of smaller bugs.

• The prefix + and - operators were not preserving integer type when performed on an integer.
• The truncate method was not returning an Int, as its signature said it should.
• Parrot's String Array PMCs were missing the get_iter vtable method to get an iterator for them; applied a patch from chromatic to add them and wrote a test.
• Made subset ... of ... where syntax without a block work (you can just mention something there that is an expression to smartmatch against, which can be a load neater for some use cases)
• Worked out what was wrong with my previous patch written over the weekend for a Parrot GC bug, and applied a corrected one, resulting in one less spectest failure and improved stability in interactive mode. Plus this helps everyone else using Parrot.
• "my Foo $x" already correctly put a Foo protoobject in $x if Foo was a class, but did the wrong thing for roles and constraints. Now it does the right thing - sticks a Failure object in there.

So, a pretty productive day, if a bit all over the place. Thanks to Vienna.pm for funding this work, and for introducing me to a nice place to eat curry in Vienna after their Monday tech meet, which I popped along to. :-)

The Perl 6 design team met by phone on 28 May 2008. Larry, Allison, Patrick, Jerry, and chromatic attended.

Larry:

• mostly off for this long weekend elsewhere
• continuing to work on quoting roles
• they need generic support to mix in roles with parameterized strings as starter and stopper sequences
• that's an interesting problem, when mapped into Perl 5
• no support for generic roles or parameterized types yet
• I'll probably just do an eval

Allison:

• reworked continuations in Parrot, based on our discussion last week
• I have very high regard for the value of reading through the commit logs
• found the source of the problem that way
• someone committed the problem about a year and a half ago
• now continuations work the way they should work
• we now don't try to make CPS respect the dynamic environment stack
• it's largely unused now that we don't store exceptions there anymore
• tracking down one last bug
• I can duplicate it
• it's an exception thrown, caught, and resumed from an :onload subroutine
• looks like they get executed differently from regular execution
• first draft of the Bylaws and Articles of Incorporation for the Parrot Foundation
• hired a lawyer in Washington state for incorporation there

Patrick:

• added quite a few things in the Rakudo implementation
• added a new metamodel that looks like the Perl 6 metamodel
• with Jonathan's and Jerry's and Moritz's help, we converted the compiler tools to that model
• cleaned up some of the inheritance hierarchy in Rakudo
• builtin types report correct methods and types
• a Parrot Integer reports itself as an Int in Rakudo, down to the metaobject information
• fixed pair handling, especially for calling functions
• people are testing those in great detail now
• now it's obvious that Rakudo's and Parrot's and Perl 6's argument handling there somewhat disagree
• doing some PGE refactoring
• primarily fixing up its class hierarchy; looks pretty easy
• may move Rakudo's grammar to use protoregexes
• might help get the metaoperators working -- French brackets, for example
• if we don't get this soon, people might submit patches doing things the wrong way (multiplying new operators for various combinations)

Jerry:

• Google Summer of Code started this week
• the aggregated feed of Perl GSoC 2008 student and mentor blogs is available
• the students will report weekly status at the #parrotsketch meeting
• the students and mentors will report at #perl6-soc on irc.freenode.net on Wednesdays at 18:30 UTC
• all of the students seem to be on track now
• made some spectest changes
• made some changes to rakudo List methods, applied some patches
• made some Rakudo test changes, should make test management easier and allowing us to create private and shared lists of tests we want to run
• should make it simple for rakudo developers to specify a subset of test files, making focused development and focused testing easier

c:

• not a lot of energy for hacking for some reason
• applied a few patches
• cleaned up a few tickets
• talked with Andrew quite a bit yesterday about GC plans
• he has some good ideas, including a clever marking scheme for which I have high hopes
• mostly on the right track
• will write up some guidelines on profiling and optimizing
• have one idea to optimize keyed object attribute access
• otherwise really need to work on the PIR profiler

Richard:

• trying to plan a legitimate way to ascertain Perl 6 grants, given the donation from Ian Hague
• planning a process around that
• thinking about YAPC::NA and a grant process BOF

Allison:

• Patrick mentioned argument passing in his report
• we don't quite understand how it needs to work in Perl 6
• Parrot will support whatever Perl 6 needs, though it may not necessarily be the default
• one goal of argument passing is to make it lazy
• you don't have to parse the whole list to start pulling out parameters
• required parameter positionals are clear
• required positional parameters are clear
• you can pass in a required positional parameter as either a positional or named argument
• Parrot maps the positional arguments to positional parameters
• when it runs out of positional arguments, it starts checking named arguments

Patrick:

• that's not really what Parrot does
• Parrot fills positional parameters with positional arguments
• then it fills in named parameters with positional arguments

Larry:

• what do you mean when you say "named parameter"?

Allison:

• pairs

Patrick:

• all parameters are named in Perl 6
• they can be filled by a named argument
• Parrot has parameters that you can fill only by position
• then there are parameters that you can fill by position or name
• but Parrot fills them opposite from how Perl 6 would expect; it chooses the positional argument over the named one.

Allison:

• I thought you needed a way to skip over a named parameter in a list
• and then fill in its value from the named argument list
• or scan the entire named argument list first, and then fill in positional parameters
• it sounds like this is different

Larry:

• the compiler should recognize arguments intended as named parameters
• it can set up a data structure for efficient access
• the other constraint is that any argument can depend on the value of a previous argument in the list
• you have to bind them in the order of declaration
• you go down the list
• for each of them, you look up in the named argument list
• to see if it's bound by a name
• if not, you take the next one from the positional list

Patrick:

• that can be lazy

Larry:

• yes
• that's the abstract view from the Perl 6 end
• one reason we invented prototypes is so that the compiler can know that the first n arguments are positional
• map any named arguments into positional without having to do the lookup
• that's primarily a sop to efficiency
• for a very small number of named arguments, brute force is probably faster

Allison:

• do you never have any parameters which are strictly only positional?

Larry:

• there's no way to write that
• other than writing a bare sigil in the declaration
• you're not giving it a name
• but you can't access it, because it needs a name

Allison:

• sounds like we want three different flags in Parrot
• one for a purely-positional argument

Larry:

• the optimizer might use that, if it sees a prototype

Allison:

• we keep the :named flag
• for named parameters

Patrick:

• named-only

Allison:

• we don't really have that
• we have named that acts like positional

Patrick:

• we have named that prefers positional, and then throws an exception if a matching named argument is also supplied

Allison:

• might have four flags then
• one that's either named or positional
• maybe one that gives priority to the positional
• not sure that's useful

Patrick:

• I couldn't come up with a case where that's useful

Larry:

• some languages want that

Patrick:

• you can almost simulate that with slurpies in your code
• or sticking in dummy arguments

Allison:

• could come in useful if you have optional parameters
• you don't have to pass anything at all

Patrick:

• I proposed what you described as numbers 1, 2, and 4
• the existing :named flag is named-only
• the :positional flag is a named parameter that accepts positional arguments

Allison:

• I'm trying to get down the ideas
• I probably wouldn't use :positional for that
• important to get the categories right

Patrick:

• I couldn't figure out a case where you have a named parameter that prefers the positional over the name

Allison:

• make it a positional and don't worry about it
• if you make it positional, you don't have to scan the argument list

c:

• look at Python's default parameter handling
• I think it may follow the one case you couldn't figure out
• not entirely sure

Patrick:

• is the answer that Parrot's underlying model will likely change to support what Perl 6 needs?

Allison:

• yes
• exact details yet to be determined
• as soon as possible so that you can use it

Patrick:

• it's been the subject of at least five or six RT tickets in the Rakudo queue
• they look like different problems, but they call come down to the same problem
• it's becoming a pain point
• I'm willing to implement it
• there are probably three or four related tickets in the Parrot queue

Larry:

• there was a question about writing Pairs various ways as arguments
• are they intended to be named arguments or pairs?
• how do you write the prototype for list, by the way?

Patrick:

• I suggested putting colons in front of it

Larry:

• but :a is always a valid term

Patrick:

• you can always look ahead for a fat arrow

Larry:

• I'd almost rather write :> for that
• originally we always used the fat arrow
• then we put in the colons
• we wanted to use the fat arrow for arguments
• people seem to be used to the colon for named arguments, not the fat arrow
• maybe we should just bite the bullet
• you need the name as the identifier
• you don't need the general pair form
• the fat arrow makes a Pair not intended for a named argument, unless you do processing on it
• that'd be a simplification
• you can tell by just looking at it
• one is a named argument
• the other is just a positional parameter

Jerry:

• is there a method on a Pair that allows it to be passed as a named parameter?

Patrick:

• prefix vertical bar?
• S06 has options for making a Pair a named argument
• or a hash as a list of named arguments

Larry:

• we're trying to guess it based on the context of the argument list
• if you pass the list to a function call, it assumes you want named arguments
• otherwise you don't
• that might be guessing too far
• the list function might want to parse more like parens
• but it behaves like a function
• you don't want to pass those as named arguments
• how do you tell the compiler that it's special
• maybe you want to make it a macro, but that's kind of weird
• maybe it's time to look at the cultural issue of how people want to write named arguments
• maybe we can simplify that

Patrick:

• there's a similar issue with the hash constructor
• you can kinda get away with it, because order is less important
• but it still exists
• it'd be interesting to look at the existing tests and Synopses to see the results
• there is some historical baggage toward using the fat arrow

Larry:

• the guy who wrote that Synopsis was familiar with Ada's fat arrow for named arguments

c:

• is there some sort of prototype declarator that says "It was an ordered list in the source code, and that's what I want!"

Larry:

• macros are a lot like prototypes

Patrick:

• Rakudo for now will treat them as special in the way that Pugs does
• it'll treat list as a macro
• do something special, it's not a normal argument list

So, after some time "on the road", mostly spent going to Perl events, I'm finally back home, where I'll be pretty much staying for the next two and a half months. Certainly, I don't plan to fly anywhere before YAPC::EU::2008; no doubt I'll enjoy a couple of weekend breaks that I can take by train first, though. Home is currently Bratislava (capital of Slovakia), so having so many interesting bits of central and east europe so close to hand will be far too much to resist. I plan to make it over to the western Ukraine (I loved Kiev and want to check out Lvov), and hopefully see Poprad and take a bit of hiking in the nice mountains in that area.

I've now got all of my slides uploaded to my talks page, and here are a few direct links with summaries to help you get to the juicy stuff faster.

• Implementing Perl 6, in Perl 6, on Parrot - I gave this one at the Nordic Perl Workshop, French Perl Workshop and to Vienna.pm. The slides were very nearly the same for all of them, so I just linked to the Vienna.pm version, which was the final one.
• Perl 6 Tutorial - from the French Perl Workshop. This was delivered in the space of two hours, and assumes no prior knowledge of Perl 6 (but some basic Perl 5 knowledge). The vast majority of what it covers is usable today in Rakudo.
• Understanding Perl 6 - this one could also have been called Perl 6 For Computer Scientists (there were variuous people who were doing academic research in computer science at the workshop). It's aimed at explaining how the features of Perl 6 fit together and giving a big picture of the language rather than dealing with syntax and practical usage.
• Using Perl 6 And Parrot In Teaching - a look at how Parrot, PCT and Perl 6 could be used in teaching.
• All Your Dynamic Languages Are Belong To Us - this was a guest lecture at a university in Stockholm to a bunch of undergraduate students. I introduced both Parrot and PCT, and then built a compiler live on stage. Overall, the talk lasted an hour; the compiler was built in just over 35 minutes.

Thanks to Best Practical and The Perl Foundation for giving me a grant to travel to help fund my travel to these events, to Claes Jakobsson for very kindly letting me stay at his place in Stockholm (and also for cooking some really delicious Swedish meatballs) and to the many people I met along the way who made the last couple of weeks a lot of fun.

Earlier today I realized that the numbers I reported yesterday for Rakudo's "make spectest_regression" were incorrect -- the Test module was not reporting the number of tests skipped (345), so the actual number of tests passed is only 430. That's still pretty good.

But on the plus side, since yesterday some excellent detective work and updating by moritz, bacek, particle, Auzon, and others means that we could add nine additional more test files into the spectest_regression target, and we're now passing 638 tests in the regression suite. Not bad for a 1-day improvement.

I suspect that there's more like this -- tests that can be moved into the "passing" category by simple fixes to the test and/or Rakudo itself. So, thanks to everyone who is helping to review the tests -- we're definitely making progress.

Pm