Rakudo Perl: November 2008 Archives

Today was my weekly Vienna.pm-funded Rakudo Day, and I spent it digging into the RT queue to see what bugs and patches people had been sending in and do some fixes and patch application. Here's what I got up to.

• bacek noted that writing "my $a; $a = $a + 1i;" led to Parrot giving us a multiple dispatch failure. I fixed this by making sure any non-Complex values were coerced before we tried to get Parrot to do the addition, so you now get the auto-vivification that you expect and this just works.
• Continuing the theme of reviewing making stuff work when you stick it inside a namespace, Chris Dolan supplied another patch, this time taking on roles. It solved half of the problem; I provided a fix to solve the other half and applied it along with some tests. Thanks to Chris for the patch and prodding me to do the other bits that were needed.
• bacek discovered an interesting bug when you tried to name an enum element the same way as certain Parrot vtable functions that I'd overridden. Turns out I forgot the anon flag, so they were added as normal non-vtable methods too. D'oh. And fixed.
• Ilya reported that trying to declare a Range type constraint on an attribute in a class led to a Null PMC Access. I was a tad baffled, until I discovered a missing null check in an attribute cloning routine.

• A ticket form masak expected the eval with a block would work. The spec says otherwise, but we were crashing in a bizarre way when people tried to use the block form - on exit! I put in a type check on the parameter to eval, as shown in S29, so it tells you outright that it's wrong now.
• Fixing this reminded me that the error message for a type check failure could be more helpful and mention which sub you had invoked when the failure happened. So I made it so.
• masak wished that '$a .= "A"' would give you a more helpful error message. In fact, the compiler was trying to, but was calling panic on something that didn't know how to. Fixed and now it tells you that you were meant to write a method name over on the RHS.
• masak also reported that .= didn't work out so well if the LHS was an index into an array (like @a[0]). It didn't work for doing the equivalent with hashes either. Thanks to PAST improvements since I did the first cut of .=, it was now trivial to refactor .= to do the right thing.
• Patrick and I spent a while discussing a couple of the messier areas of actions.pm and worked out ways to improve them. There's tasks for both of us in there. We also talked about how to implement context variables, and I think Patrick plans to hack on that soonish. Those will be cool to have. :-)
• Fixed a bug reported by masak relating to lexicals with the & sigil, and added a couple of spectests to cover it.
• Moritz pointed to an integration test that he'd added, known as the man or boy test. Rakudo turned out to be a bit of a boy, but a couple of fixes later it's now a man. (Fixes were to make parameters with the & sigil work properly - they were meant to but there was an small bug - and fixing up 'is copy' a bit. We run through the first 10 values, and then hit Parrot's developer-aiding recursion limit in the 11th. But computing the first 10 pretty much shows we can do it. I pondered upping the limit for Rakudo (a default value of 1000 calls deep is compiled into Parrot, but it's changeable at runtime), but the limit is good for now while we're in development. Better to catch infinite recursions early.
• Ilya discovered a bug that was introduced when we refactored containers. It showed up when you passed an array and then iterated over it. I put in something that I thought would fix it (it fixed every test case in the email, all of spectest that we were passing before and some tests from the email that Moritz had put in a test file). However, then Patrick pointed out a case that it didn't work for. No good solution yet, but I added his test case to the spectests so we don't forget to address it. In the meantime, hopefully this makes things a little better for Ilya.
• Responded to and/or commented on a few other tickets.

So, a day full of bug fixing rather than doing shiny new features. Which I guess makes not so exciting reading, but I hope this contributes to a more stable Rakudo and a better experience using it. Plus there's tests to cover the issues that were reported and fixed, meaning we've got better test coverage and some hope of not regressing on any of these in the future. :-) Thanks to everyone who provided the feedback so we knew these bugs existed, and to Vienna.pm for funding me to do the fixes.

I'm working hard at the moment on getting my long-overdue MMD grant from DeepText finished up. I've got some various bits of progress to report.

First, I cleared up the subset type code. The first cut I did of it a while back had a Subset class, which shouldn't really have existed. Now a subset is (under the hood - *not* conceptually) an anonymous subclass of the thing it is refining which overrides ACCEPTS and keeps the block around. The Subset type is gone. Also, importantly, it's easy to now find out what the real, non-subset type at the "root" of it all is. And with that in place, I could get subsets and their interaction with nominal types correct in the multiple dispatch algorithm. So, that should now work just fine.

I also got the "default" trait in place. This means that if you have two candidates that could be ambiguous, you can mark one of them as the default. A (useless) example is:

multi foo() { say 1 }
multi foo() is default { say 2 }
foo();


Which will always output 2. As I said, useless here, but if you have a bunch of maybe-not-quite-mutually-exclusive-always subset types and want to mark one as the default if you know there's a potential overlap this is a way to do so. And with this in place, we now have all of the ways that a dispatch is decided implemented. To the best of my knowledge, it's correct.

I'm generally not into performance work at this stage of the game, but MMD is heavily used in Perl 6. In the not too distant future we will have a prelude written in Perl 6, and then all of the operator dispatches will happen through the Perl 6 dispatch algorithm. Further, Parrot's MMD for ops had recently got more costly as it got unified with the MMD for subs (good in that we no longer have two different multiple dispatch algorithms in the Parrot core). Thus I set out to write a basic MMD cache that would aid both of these.

I didn't optimize the heck out of the cache, I just did it quickly with what was easily to hand. The same caching system can be used by both Perl6MultiSub and Parrot ops, and in the future the Parrot MultiSub PMCs too (which the compiler users, so we may get a small performance gain there, but it won't be much). I got Perl6MultiSub using it and, after fixing a bug involving arguments instantiated from anonymous classes, all tests passed with the cache. The performance was an improvement. I ran as a benchmark:

multi sub foo(Int $x) { 1 }
multi sub foo(Num $x) { 2 }
loop (my $i = 1; $i < 25000; $i++) {
    foo(4);
    foo(4.2);
}


Without the cache, it takes 20.25s on my box; with the cache it runs in 6.953s. Note that only part of the time in the execution is spent in the dispatcher, and it takes well over 3 times improvement in dispatch time itself to make the whole program run in less than a third of the time. So, that's a satisfying improvement (though looking at the times, we're still only making 3500 calls to a multi sub per second here...then, this is an unoptimized Parrot build and the Parrot calling conventions that actually do the args passing are a known area in need of optimizing, plus we're waiting on GC improvements too, so I'm optimistic that once the other factors improve we'll be doing better).

So how does the cache actually work? We cache on the types of the arguments. This involves collecting their type identifiers and, for now, building a hash key out of them. In the future we may be able to do something smarter. But it means that if you have the operator infix:+ and it has been overloaded plenty, and you're calling it repeatedly with a type it hasn't been overloaded for, you'll now just hit the cache each time after the first call, rather than having to run through a bunch of type-narrower candidates that come before the more general candidate.

Note that, due to a few Parrot test fails we're tracking down in the use of the cache with MMD'd Parrot ops (not directly a problem in the cache itself, but providing the right info to the cache), it's currently in a branch. All Rakudo's tests and the same bunch of spectests are passing with the cache, however, so Rakudo is ready to run with it. We'll get it merged soon - maybe even later today or tomorrow.

Things coming very soon: making multi methods work as well as multi subs with the new dispatch algorithm and supporting the double semicolon to specify some parameters are not part of the long name and should not be dispatched on. Thanks to DeepText for funding this MMD work.

On behalf of the Parrot team, I'm proud to announce Parrot 0.8.1 "Tio Richie." Parrot is a virtual machine aimed at running all dynamic languages.

Rat Creature #1: Comrade! We are about to feast! Quick, get your fat carcass behind this bush and get ready!

Rat Creature #1: Hello, small mammal.... Could you step in here for a moment? I've got something to show you....

Fone Bone: Can't you show me out here, where I've got runnin' space?

Rat Creature #1: No! No! Please! Step in here -- your friend the dragon isn't around, is he?

Fone Bone: Hey, Ted! Where you goin'?

Ted: You're on yer own, Bone! (exit stage left)

Rat Creature #1: Quick, comrade! Start the cooking fire!

Rat Creature #2: No. You called me fat.

Fone Bone: Ted! Wait for me!

Rat Creature #1: (pulls Bone into the bush) Well, well... Look who's joined us for supper... Go start the cooking fire!!

Rat Creature #2: No. You called me fat.

Rat Creature #1: No?!!! What do you mean, no?!!!

Rat Creature #2: And it's not the first time you've done it either....

Rat Creature #1: Comrade... be reasonable! I wasn't thinking -- I was trying to catch our dinner -- this isn't the time -- I take it back.... you're not fat.

Rat Creature #2: Too late!

Rat Creature #1: Please, comrade! I just want to chop him up for the stew!

Rat Creature #2: And that's another thing. I'm tired of stew! I want to put him in a crust and bake a light fluffy quiche!

Rat Creature #1: Quiche?! What kind of food is that for a monster to eat?! .... Listen, do you think you could come back in half an hour? We'll have this straightened out by then!

Bone: (runs away)

Rat Creature #1: (beat) Why didn't you stop me?

Rat Creature #2: Why should I? You're so smart!

time passes

Fone Bone: (hanging from a branch in a waterfall) Those rat creatures would have to be pretty stupid to follow me on to this frail, little branch!

Rat Creatures: (follow Bone on to the frail, little branch)

Fone Bone: Stupid, stupid rat creatures!! (branch breaks)

— "Bone", by Jeff Smith

Parrot 0.8.1 is available via CPAN (soon), or follow the Parrot download instructions. For those who would like to develop on Parrot, or help develop Parrot itself, we recommend using Subversion on our source code repository to get the latest and best Parrot code.

Please note the updated Parrot Roadmap and our goals leading to the 1.0 release.

Parrot 0.8.1 News:

- Implementation
  + added CPAN module Storable 2.12 as a configuration and build dependency
  + removed the pseudo PIR opcode 'addr'
  + added the 'box' opcode
  + fixed 'pop_eh' handling in PIR libraries and examples
  + removed usage of .return for tailcalls (use .tailcall instead)
  + removed 'get_hash' and 'get_array' from Capture PMC and Capture_PIR
  + improved debugger and HLL coordination
  + allowed MMD primitive autoboxing
  + fixed all known memory leaks in PIR "Hello, world!"
  + NCI signatures now JITted on x86-32 platforms (Windows and Linux)
  + made the .const directive take a quoted type name instead of a constant
  + made IMCC more re-entrant
- Languages
  + Rakudo
    - refactored Junctions implementation
    - added "fire and forget" tool to rebase/rebuild/test Parrot and Rakudo
    - updated container/reference semantics
    - added more builtin methods and functions
    - improved support for multilevel namespaces
    - added support for .Str, .succ, .pred in user-defined classes
    - implemented pointy blocks on if/loops
    - increased STD.pm convergence
    - added %*VM hash
    - improved MMD candidate sorting
    - improved integration of Num and Int
    - implemented increment on protoobjects
    - added initial support for MAIN subs
    - added .PARROT method
  + Pipp
    - added some predefined constants.
    - added implemention of the function basename().
  + Cardinal (Ruby)
    - Added initial support for the classes Proc, Continuation, Queue, Dir, File, and FileStat
    - fixed various minor bugs
    - fixed broken Regexes
- Compilers
  + PCT
    - added '.isa' method to PCT::Node
    - cleaned up 'immediate block' handling
    - allowed arguments to immediate blocks in loops/conditionals
    - metaclass objects can now 'add_method'
  + PIRC
    - integrated macro processing in PIRC's lexer
    - integrated heredoc lexer in PIRC executable
    - added preprocess and heredoc-preprocess commandline options
    - integrated the PASM grammar into PIRC
    - added a register allocator to optimize the built-in vanilla allocator
    - code cleanups and documentation
    - added 'make test' target
   + TGE
    - updated to work with new namespace/classname syntax
- Deprecations
  + PARROT_API will become PARROT_EXPORT
  + :lexid will become :subid
- Miscellaneous
  + Documentation
    - updates and additions to the Parrot book (see docs/book/)
  + OpenGL
    - minor fixes to Parrot's base OpenGL bindings
    - converted OpenGL PIR examples to Perl 6 (two complete, one WIP)
    - added simplified OpenGL PIR example for NCI JIT testing
  + Tests
    - further improvements to parallel tests
    - tests no longer leave filesystem breadcrumbs

Thanks to all our contributors for making this possible, and our sponsors for supporting this project. Our next release is 16 Dec 2008.

Enjoy!

Tonight I updated Rakudo's ROADMAP file to reflect the progress we've made since it was last done (August 2008, YAPC::EU). At first I thought there wouldn't be much to update, but it turns out we've accomplished a fair bit.

More importantly, more infrastructure pieces are falling into place so that the remaining items become even easier than they were before. I expect to see good progress during the next month or so.

This weekend many of us attended the Parrot Developer's Summit to plan the way forward to Parrot 1.0 (March 2008). In parallel several of us also made plans for Rakudo's development in that time -- our goal is to have Rakudo on its own release cycle by then. I very much like the way the time-based monthly release cycle has worked for Parrot development, so we will undoubtedly continue this in Rakudo. Development for Rakudo will continue to track the Parrot trunk for a while, and we'll time Rakudo releases to follow a few days after each Parrot release.

We'll put together more details as we get closer; until then we're continuing to work on feature development.

During my airplane travel and at the summit I was able to get a lot of work on fixing lexical variables in Parrot. Unfortunately I've also found several places where Parrot was not properly allocating and freeing contexts, so those need to be reviewed and fixed before we can merge the branch back into trunk. Initial tests show that the design itself appears to work as expected, so now it's just a matter of reviewing and repairing context handling in Parrot. I'm expecting (hoping) to merge the branch back into trunk sometime on Wednesday (after tomorrow's Parrot release), which will close out quite a few RT tickets.

Pm

The Perl 6 design team met by phone on 12 November 2008. Larry, Allison, Patrick, Jerry, Will, Nicholas, Jesse, and chromatic attended.

Allison:

• we have two sponsors now, BBC and ActiveState
• starting conversations with a few others
• it helps to have two already
• mainly focused on the IO milestone milestone
• will merge back in two stages
• stage one is getting the new IO system working
• you can use it as an object
• adds just one test file
• I can have all of the different architectures report any failures with that test file
• will merge in the bigger change later
• though it's a simpler change
• replace calls to the existing system with calls to the new system
• hope it'll be a smooth transition
• fixed up some calling conventions stuff last night for Andrew Whitworth
• some pieces weren't working for him
• committed a small patch to fix them, and explained it

Jerry:

• the first Parrot Developer Summit is this weekend
• for people who don't know about it, it's too late to attend
• there may be a dozen or so of us meeting
• intent is to hammer out a roadmap for 1.0
• lots of questions on the channel and mailing lists about when there will be Parrot 1.0
• good, I suppose

Patrick:

• Christmas Eve!

Jerry:

• they've given up asking about Perl 6

Jesse:

• good!

Jerry:

• I don't have an answer yet
• but our work this weekend will let us decide what we need to release 1.0
• also had a grant approved
• designing and implementing Perl 6 command line syntax
• already started the work
• researching other dynamic languages and compilers
• most languages have just grown their syntax over time
• it's kind of messy
• have put together some ideas
• discussed them with Larry
• expect that to continue, until I have a design document that I can send for review
• then I'll move on to implementation
• all laid out in the grant proposal on TPF's site

Jesse:

• expect me to harass you occasionally

Patrick:

• things are going well
• lots of additions to Rakudo this week
• I fixed the string semantics
• classes can override how objects report stringification
• Jonathan, Jerry, and I fixed Rakudo's container semantics
• it does references correctly
• fixed a lot of bugs and closed a lot of tickets
• Jonathan started and I finished getting pointy blocks to work on flow control statements other than for
• waiting for answers on repeat while and repeat until

Larry:

• me too

Patrick:

• when one of us gets the answer, he should let the other one know
• completely refactored Rakudo's junctions
• worked on the assignment metaoperators, such as += and *=
• work correctly on protoobjects
• I'm refactoring those now
• will cause regression in one test that doesn't work right anyway
• Rakudo is now over the 4500 passing test mark
• we should get quite a few more over the next week or so
• Jonathan and I will continue closing things out
• worked more on the design of lexicals
• continuing to update the document on my website
• need to figure out autoclose in my head
• Jonathan and I will hack on that this weekend
• I could use his expertise in Parrot internals
• also had a grant approved

Larry:

• went to the hackers conference this last weekend
• high point was teaching Don Knuth how to decode his Korean email

Jesse:

• I thought he had no email!

Larry:

• refactored the compile-time variables
• there are no version-specific ones remaining
• now $?OS, $?DISTRO, and $?VM variable
• generalized matching against pairs to call a method corresponding to the name on whatever the object is, matching against whatever the argument is
• sort of inside out as to what named arguments in BUILD routines do
• refactored the :pair-named methods
• they were kind of grungy anyway
• thinking about whether strings ought to have methods like .r and .w and .x
• or whether you ought to coerce to some sort of File type

Patrick:

• I lean toward coercion

Larry:

• everyone does
• I just want to think it through
• decided that .perl isn't special to junctions
• if you use it on one, it autothreads
• if you stringify that junction, it produces a result as if you'd done .perl on the whole junction
• stringification on a junction produces Perl-style syntax
• if you smart match a hash against a pair pattern, it tests the key and the value for matching
• factoring out various amounts of boilerplate from the Perl 5 emitted code
• simplifying code
• fixed bugs with node wrapping
• confused people looking at ASTs
• chopped out TRE
• discovered that my stupid LTM emulation ran twice as fast as the TRE code
• that was eye-opening
• STD now recognizes ::T as defining a valid type for later in the same scope
• did some work to set up the infrastructure for STD to handle lexical scopes
• nested pads
• should have declarations in the right scope
• part of the motivation
• besides better error handling
• set up a mechanism to define a prelude
• freeze the lexical environment for the prelude
• then slurp that up into STD
• mundane functions and operators will probably end up there
• have most of an implementation of a trie engine
• that'll replace my stupid LTM
• should evolve eventually into a full DFA
• almost works
• should run faster
• and cut down on the on-disk cache
• for various definitions of "should"

c:

• mostly fixing bugs
• trying to keep blockers from blocking people
• adding a little polish here and there to make people work better
• thinking about our release process post 1.0
• will talk about that this weekend

Will:

• spending most of my time deleting things from Parrot
• ripping out deprecated features
• everything that's in there, some goes back to 0.5
• but everything in there now we can rip out before the next release
• sad that I'll miss the summit this week

Nicholas:

• I'm the new grant manager for Dave Mitchell doing 5.10.1
• announced the end-of-life for 5.8.x
• surprised that no one told me I couldn't say that
• if the Parrot Foundation is capable of raising money from people, how come TPF isn't?

Allison:

• I tried to do something similar for TPF four years ago
• it was very uncomfortable for the Perl community
• lots of pushback
• they wanted guarantees that no company would have control over the development of Perl

c:

• no one has control over the development of Perl!

Allison:

• also the amount of money required for Perl 5 is much larger
• it's not about money for Parrot
• four memberships for Parrot is enough to push toward 1.0
• we're a small project

Jerry:

• have Perl 5 releases been sponsored before?
• seems like a good thing

Jesse:

• last week in China for the Beijing Perl Workshop
• it was a lovely workshop
• chatted with a lot of people about the state of Perl 6 and how they could get involved
• offered to make lots of introductions
• haven't taken anyone up on that yet
• the Perl problem in China is that they're having a hard time finding competent Perl hackers
• people are afraid to learn Perl because that'll make them unpalatable to the workforce
• universities only teach C
• people study on their own to learn C# and Java
• most of the hiring firms really want C# and Java from their developers

Patrick:

• in Rakudo, over the past week, I'm seeing lots of segfaults in different places
• suspect it might be something that Jonathan and I did
• have no clue where to start looking
• it shows up in the regression suites
• can point you to specific instances of things that fail

Will:

• I can help narrow it down to PIR if you have a specific test that fails

Patrick:

• they really float around
• showed up in the assignment tests this morning
• a couple of revisions later, they don't show up later
• only some of them does -G help
• the weird one this morning
• if I run it from Parrot with the command line, I get a segfault
• from the test harness, I don't
• that seems backwards

Allison:

• the test harness has more strict flags turned on

Patrick:

• at one time, it used a different runcore

c:

• if there's a short PIR test case, we may be able to do a binary search

Patrick:

• it's hard to give a short PIR case
• the segfault moves around so much

Today was this week's Rakudo day, but between this and the previous post I'd got a bit of hacking done too. During a mini-hackathon at my apartment after the excellent Twin City Perl Workshop, I got a first cut of the 'is also' trait for classes implemented. This means that you can add extra methods to an existing class (you will be able to do this with attributes in the future, but that's harder to implement).

class Foo {
    method test1 { say 1 }
}
my $x = Foo.new;
class Foo is also {
    method test2 { say 2 }
}
$x.test2; # 2


The thing that prompted me to implement this is that we want to start writing many built-ins in Perl 6, but will also want to keep some bits in PIR. We should only define the class in one place, and then add to it in another. This change will allow us to do this. Hopefully it's the last blocker to getting a Perl 6 prelude - Patrick is, I believe, going to be trying this out pretty soon. Today I made sure we had a regression test running for this feature.

Afterwards, I looked at sorting out one of our other long-standing issues: you could not write:

my Num $x = 42;


Because it would fail with a type check error. Num was supposed to accept Int values. I fixed this, but the regression tests then started failing for multiple dispatch - an area I'm still actively working on. I boiled the problem down to being the same issue with candidate sorting that was plaguing other code, and dug in to taking a look at what was going on. Some stepping through with the debugger late, I soon found what was going on: we were removing edges from the type narrowness graph too early, and depending on the order in which multis were defined could end up considering two candidates as tied when producing the dispatch order when in fact the type narrowness analysis had determined otherwise. A fix to this later, and that (and no doubt other issues) were resolved. Tests passed, and the patch went it.

I checked on another multiple dispatch ticket quickly, to see if that one too was now resolved. The issue mentioned in the ticket was fixed - at least, it was when I ran it in the REPL, so I marked the ticket resolved. Then I realized the applicable spectest was still failing. A little digging later, I noticed that we were creating subset types a bit too late, moved them to the same time we created classes (which isn't right yet, but that's a harder problem to tackle). I found another incomplete bit of the multi dispatch stuff along the way, but decided to leave that for another time (it's still something of a work in progress). Anyway, long story short, this much now works (and is spectested):

subset Even of Int where { $_ % 2 == 0 };
subset Odd of Int where { $_ % 2 == 1 };
multi sub test_subtypes(Even $y){ 'Even' }
multi sub test_subtypes(Odd $y){ 'Odd' }
say test_subtypes(3); # 'Odd'
say test_subtypes(4); # 'Even'


I was happy to have another patch from Chris Dolan, which fixed up another place where we had a problem with nested namespaces. Applying the patch, however, caused a bunch of spec tests to fail. I was surprised, since it looked good to me. Digging deeper, it turned out to be a code generation issue in PCT. I resolved that, and then the patch worked just great. :-) While doing that, I noticed that there had been some partial breakage to grammars during some other changes. I fixed that too, then made sure we had test coverage for both of these issues, so they don't come back again.

It's always nice to see people playing with Rakudo and doing stuff. japhb has now got some OpenGL things working in Rakudo, and was interested in having the MAIN sub supported. The idea of MAIN is that, if the program is run from the command line, this is the entry point. There's some really cool stuff that happens in terms of parsing command line arguments and turning them into named parameters - indeed, Jerry Gay aka particle has got a grant to work on this, so I didn't dig into that. Instead I just pass them as positionals to MAIN. It's a start, and you should have all of the goodies as Jerry reaches that part of his grant in the next couple of months. :-) As a taster:

sub MAIN($first, $second) {
    say $first + $second;
}


When invoked as "perl6 test.p6 7 35" outputs 42.

Finally, I did a first cut of $?PACKAGE, a variable that tells you what package you are currently in.

Thanks to Vienna.pm for sponsoring today's work.

With some Rakudo days to catch up on post-vacation, and knowing Patrick was keen to get some container changes rolling, I made a second Rakudo day this week on Thursday. While waiting for Patrick to show up (we're quite a few timezones apart), I hit the RT queue.

• I got an easy start to the day - finding a ticket that covered an issue pmichaud++ had resolved already and closing it. The issue in question was to do with making objects stringify to something rather than giving an error - this now happens. Further, you can write an Str method on a class that returns a string now to override this behavior.
• Here's teamwork. Last night, Carl Masak reported a bug when you wanted to inherit from a namespaced class. I said I'd look at it today. In the meantime, Moritz got some unit tests written. I went to the ticket in RT today to find that Chris Dolan had already sent in a patch to resolve the issue, leaving me to review it, test it and apply it.
• Spent quite a while trying to get if blocks that take a lambda to work (the idea is that you can capture the expression that was evaluated as the conditional). Got close, but something was messed up in the code generation. Then put it aside to work out the container changes.

Once Pm appeared, we dug into the refactoring, in a branch. The aim was to fix up various container semantics, and do various things correctly that we weren't doing before. As a side-effect, it would also aid memory usage. I started doing the big changes and got us down to failing under 2% of the regression tests that we passed before starting the refactor in just a few hours, and had various "unexpected successes" too (where TODO tests start to pass). I then worked with Patrick to track down some of the others, before finishing up my talk for the weekend's Twin City Perl Workshop and getting some sleep in before that. When I checked in on the IRC channel the next day, Patrick had finished and merged the branch. Over the coming days we'll probably find some more tests that we can now un-skip as a result of these fixes.

Thanks go to Vienna.pm for funding Thursday's work, and being a big part of making Twin City Perl Workshop a really good one.

The Perl 6 design team met by phone on 05 November 2008. Larry, Allison, Patrick, Jerry, Will, Nicholas, and chromatic attended.

Larry:

• the specs officially define a Nil type that is the meaning of ()
• an undef that's undef in item context
• otherwise, just a null list
• just to clarify the semantics of what it means to say return;
• answered Patrick's email about autothreading of junctions over method calls
• I think that doesn't have much impact on specs
• we can clarify if necessary
• in terms of hackery, I've done a lot of work on cursor and STD and gimme5
• optimizations and simplifications
• preparing to get rid of TRE in the lexer
• threw a lot of stuff out of cursors themselves
• they no longer track the original string or positional memos in the Perl 5 version
• nor do they distinguish a to attribute anymore
• an unbound cursor only as a position
• a bound cursor has a from
• for ratchety kind of matching, all it does is modify the position in a mutable cursor
• rather than copying cursors, which is fairly expensive
• a little more efficient
• made the tryfile syntax checker work on multiple test files
• keeps it from having to recompile the standard grammar on every test
• it does leak rather badly
• you can't parse the entire test suite with one invocation, unless you have a lot of virtual memory
• refactoring the various methods which determine cursors to try to make decisions earlier
• don't have to make extra function calls to make a decision which should be made at a higher level
• some speed gains, some preparation for lexer changes
• another preparation is to start emulating NFG semantics
• when I set up for the original parse, I now do an unpack "U*" as an emulation of NFG
• one integer per character
• some of my tests in my regex emulation now test against that array
• rather than against the original string
• it runs no slower, even accounting for the conversion
• eventually the lexing I want to do with a homebrew DFA matcher will use that integer array directly
• probably also buys some efficiency
• Perl 5 is not very efficient about recalculating positions
• it only caches a few UTF-8 positions for a string
• if I'm bouncing back and forth between the beginnings and endings of tokens, it has to keep recalculating UTF-8 from one direction or the other
• the net NFG will be a win
• even if some tests need to use the string version
• still planning to get rid of TRE
• the workarounds cause the lexer to take half a GB of disk space
• hope it'll run faster too

Patrick:

• worked on exceptions in Parrot
• mostly identifying places where the spec needs work, or how they don't work as I expect them
• answering questions about the specification
• worked on lexical variable handling in Parrot
• tomorrow, Jonathan and I will switch over to the new container/value semantics so they work properly

c:

• does that fix list assignment?

Patrick:

• it's necessary, but not sufficient
• PCT also needs some changes

Jerry:

• worked on some refactoring of Rakudo's action grammar
• action methods
• there were many different types of calls into the metaobject
• it was quite a mess
• I implemented an isa method on PCT's Node class
• converted Rakudo to use that instead of .WHAT or .HOW.isa
• definitely more refactoring to do in the action methods
• I'll attack that as I can
• Jonathan made some changes to Rakudo's use
• instead of compile time, it happens at INIT time
• precompiled modules work now
• but it's still a hack
• use shouldn't be done at INIT time
• his changes necessitated changes to my import patch

Patrick:

• that was the case anyway

Jerry:

• I knew that was the case
• I just need the time to rework that
• still preparing for the Parrot Developer Summit
• again, that will continue until after it happens

Will:

• did some Parrot work this week
• fixed up the sub-level tracing that's been broken for some of the more complex code
• turned out to be a cut-and-paste job from something Jonathan had done for Rakudo in Perl 6
• trying to go through the deprecated list, getting rid of some low-hanging fruit
• deleted quite a bit of code this week
• looking at the tracing stuff to help me figure out where Tcl is spending most of its time
• we seem to be invoking PGE a lot
• looking forward to some of the work Patrick has on his plate
• hopefully that'll get Tcl and Perl 6 running faster

c:

• I fixed some bugs
• fixed some crash bugs
• found some new crash bugs and fixed them
• also trying to remove deprecated code
• tracking down some other crash bugs, but they're tricky
• cleaning up code is a priority there
• still thinking about how to do PIR-level profiling
• we sorely need it

Nicholas:

• the release candidate didn't get out last weekend
• still might not get out this weekend
• waiting on a CPAN release of File::Path

Patrick:

• remember that there are two init times
• there's the semantics of Perl 6 INIT block
• there's the Parrot sub :init as well

Jerry:

• I meant the Perl 6 version

Patrick:

• there's nothing prior to :init in Parrot terms, if you're talking about Parrot
• we have to keep that in mind

Jerry:

• Jonathan pushes onto $?INIT in the Perl 6 terms

Patrick:

• it has to happen earlier than that

Nicholas:

• Larry, what version of Perl are you using, in terms of UTF-8 slowness?

Larry:

• 5.10
• it does some caching of pos
• but if your lexer runs back and forth to test the same thing various times, it still has to do some stuff

Nicholas:

• it ought to be able to store three or more positions
• might be as simple as a #define
• I haven't done that

Larry:

• I'm moving away from the whole UTF-8 model anyway toward the NFG model
• that unpack is amazingly fast
• I thought I'd lose some time, but it was just as fast
• maybe that's some kind of indication that we're going the right direction
• I'm not actually doing NFG, just assuming I'm in NFC form
• NFC is a subset of NFG

Jerry:

• with this approach, will you always need a hybrid solution?

Larry:

• it depends on what your VM is

c:

• we can recommend one for you

Larry:

• some things are easier to check in Perl 5 with the string matching
• no decent way I know of to ask whether a particular character number matches a particular character class
• I'd have to convert it to a string, then do a pattern match against it
• as far as I know
• unless you're aware of a way in which of saying "Is character #582 an alphanumeric?"

Nicholas:

• I assume you can hijack UTF-8 swatch caches to figure that out in Perl 5

Larry:

• it would take an XS hack right now
• don't think those are exposed at the integer level

Nicholas:

• part of that is implemented as Perl 5
• converted to XS, at least the slowest parts

Larry:

• I doubt it would be difficult to make an API for direct access in a version of Perl
• it wouldn't be in the current version

Nicholas:

• not likely to be sufficiently faster, when you account for developer time

Larry:

• essentially, I'm matching against UTF-32

Patrick:

• do we have a good idea or list of what methods are defined on junctions?

Larry:

• there's probably a secret method name which does the autothreading, which Object delegates to
• other than that, maybe some methods to extract out the bits of it
• they would not have to be human readable
• I'd de-Huffmanize those names to prevent accidentally running into them

Jerry:

• this came up with regard to any container, not just junctions
• how do you call a method on a container, not something that hypers over the values?

Patrick:

• it's not automatic, as it is with junctions
• the method that kept coming up in my mind is .values to grab the values out of the junction

Larry:

• there'll be a method to do that, but not with that name
• .junction_values
• something unlikely to collide with anything else

Patrick:

• there was a discussion about stringification being made special somehow
• the printed stringification and default stringification might be different
• the .perl method on a junction might return a list of strings, rather than a string
• how do we get a string that contains the junction operator in the middle, rather than a junction of two strings
• has that made it into Synopses somehow?

Larry:

• a multimethod which converts a junction to a string will fire directly without autothreading
• Str(Junction)

Patrick:

• it's a method, not a function?

Larry:

• it doesn't really matter, as long as it's defined within Junction

Patrick:

• Junction.Str always gets that version, and does an autothread?

Larry:

• if a Str method is defined in Junction then it won't autothread
• only autothreads if Object gets the method dispatch.
• always produces a string which looks as if it came from .perl
• everything falls out of the current mechanism
• we don't have to define special cases

Patrick:

• one more question
• what is the syntax for matching a string or object against a specific rule within a grammar?
• a lot of people have smart matched against the name of the rule

Larry:

• underneath, you create a match
• then just call a method on that
• the method is the rule
• how much syntactic sugar we want to wrap around that, I'm not sure

Patrick:

• can I tell people that they're doing it wrong?

Larry:

• it probably is, yes
• it sounds wrong, without seeing it

Patrick:

• essentially $variable ~~ Gramar::Rule_Name

Larry:

• that seems unlikely to work without someone deciding to create the match
• it's possible the smart match operator could be smart enough to do that

Patrick:

• if I do $variable ~~ sub

Larry:

• that depends on the arity of the sub
• similar distinction
• "Oh, this is actually a method"
• presumably of zero arity, except for the invocant
• essentialy the string has to be passed in as the invocant

Patrick:

• that is one of those things where the smart match is macro-like
• it recognizes that it's a sub, not a list operator

Larry:

• it recognizes a routine of some sort
• it needs to know that it's a method from just inspection
• just naming a method doesn't give it enough information
• unless you put it in method form...

Patrick:

• leave methods out of it for now
• does that syntax check the arity of a sub against something
• or does it treat it as a list op?

Larry:

• it'll call the sub with the left-hand argument as the first argument

Jerry:

• as opposed to $variable ~~ &sub

Patrick:

• normally a sub name in a term position is a listop

Larry:

• seems like it still is
• I don't think it'll work without the ampersand
• you have to allow a general expression on the right to return a value to smart match
• you need explicit curlies or the ampersand

Jerry:

• in the recent past, I remember discussions with the PPI folks that Perl 6 can't parse Perl 6 any better than Perl 5 can parse Perl 5
• although things aren't perfect with the parse yet or STD
• and we aren't using BEGIN and macros yet
• azawawi has created a syntax highlighter based on STD
• it's really cool
• feather has syntax-highlighted versions of all of the spec tests
• they're linked from spec.pugscode.org
• you can see syntax-highlighted Perl 6 today
• hasn't been reported anywhere
• definitely worth mentioning
• effectively a built-in