chromatic: May 2008 Archives

The Perl 6 design team met by phone on 21 May 2008. Larry, Allison, Patrick, Jerry, Will, and chromatic attended.

Allison:

• working on the exceptions migration
• finished the last half of that
• debugging failing tests now
• I have eight failing tests
• narrowed it down to three basic problems
• had a question for Patrick on that
• continuations taken on labels in the middle of a subroutine make very poor exception handlers
• independent subroutines work much better
• the problem is that setjmp/longjmp isn't stable or guaranteed to be stable in the spec

c:

• that only surprises me if you're using a bad platform with an awful compiler

Allison:

• longjmp isn't guaranteed to restore all of your variables properly
• we're seeing memory corruption when we restore
• we're ready for language testing on that branch
• the places where this is a problem is throwing exceptions from C
• throwing exceptions from within PIR is fine
• it just inserts an exception handler as the next op in the runloop

c:

• released Parrot 0.6.2 yesterday; seems fine so far
• improved performance in Parrot yet again
• ran out of time over the weekend to work on concurrency
• will apply open patches and check for bugs this weekend
• scheduled my trip to YAPC::NA, will be at the Parrot hackathon both days but not the post-conference hackathon

Jerry:

• did some work on mod_parrot on a branch
• rewriting the configure system to improve portability
• some good progress there
• added a couple of Perl 6 spec tests
• modified a few to make them pass on Rakudo
• talked to Moritz about a make target for just passing spectests
• it's make spectest_regression
• should get you noiseless, all-passing spec tests
• they run through fudge for now, but eventually we'll take that out
• trying to get back into Perl 6 development, but I can't keep up with Jonathan and Patrick

Patrick:

• now you know how I felt!

Jerry:

• I'll stick with the tests
• GSoC starts in earnest within a couple of weeks anyway
• want to make sure that all of the students have what they need

Larry:

• took a side trip to Japan
• gave my talk there on the current state of the parser
• why and how we're making it extensible
• continue to work on the parser to make language derivation more real
• revamped the quoting to use real sublanguages and get all the mixins to work right
• I Moose-ified the STD5 implementation
• now does mixins correctly
• now parsed double-quotes for escape sequences and properly returns a Match object
• now working on regexes as a sublanguage
• thinking how terminators match
• the old recursive-descent parsing was kind of brittle
• switched it to use the operator precedence parser
• figured out that I could reuse the same one by supplying different term and infix expectations in the subgrammar
• that effectively hides the parts of the regular grammar that you don't want to see
• you don't have normal regular expressions popping out where you don't want them
• there's no reason not to use the regular operator precedence table for that
• don't have to worry about propagating terminators down so that they stop on double vertical bars

Patrick:

• PGE does the same thing

Larry:

• it looks cleaner
• but it's untested
• if the operator precedence parser works on regular Perl, it'll work on the real thing
• the benefits of longest-token matching still work too
• there are occasions where I don't want to derive a sublanguage just for a different quote terminator
• playing with a single hard-wired stop pattern passed in from the dynamic scope
• may need a start pattern so I can have nested brackets
• should suffice for quoting needs without multiplying grammars for every different delimiter
• instead of the preprocessor rewriting protoregexes, the standard grammar looks at protoregex data and rewrites it on the fly

Patrick:

• had end of semester grading and stuff
• that distraction is gone
• created a new metaobject model on top of Parrot
• makes Parrot objects act like Perl 6 objects — .HOW and .WHAT methods
• follow Perl semantics, not necessarily derived from a common base
• gives a nice consistency to the toolset
• PCT and PGE use that model now
• that went quick
• Jonathan and I are converting Rakudo to that model now
• finding other updates in the process
• seems to be going well
• closing tickets, applying patches, answering questions
• figured out an approach to solving scalars in Rakudo
• will help with list assignment and list context
• a new PMC type for the base class of mutables
• we can attach properties to that
• figured out how to clean up class relationships in PGE
• need to refactor PGE
• that helped me figure out how to implement protoregexes
• that'll help us implement more of the standard grammar
• not a complete longest token matching implementation
• but it'll get us closer
• need to refactor the operator precedence parser slightly
• needs more introspection to make scalar assignment propagate context appropriately
• not a minor refactor; PGE uses operator precedence parser to parse things
• a fair amount of changes to PGE over the next couple of weeks
• I don't expect any external impacts
• our abstraction levels seem good to mitigate that
• it's easy to change the world underneath without having to change things externally
• no blockers at the moment, other than sleep

Will:

• keeping an eye on the queue
• trying to stay active on IRC
• trying to get some people fill out CLAs and get commit bits
• want to make sure we keep good patch submitters happy
• some grant committee work this week, nothing particularly Perl 6-y
• blocking on time

Allison:

• how much do you depend on exception handlers being set as continuations within a subroutine?
• how much would it hurt if they were separate subroutines?

Will:

• I'd have to update a lot of code in Tcl
• that wouldn't kill me

Patrick:

• I like the way it works now
• there could be reasons why that's not fully desirable
• but it's really nice to have all of my registers in scope with labels as it is now

Allison:

• at the moment, exceptions thrown within C require setjmp/longjmp to get to where we can insert a new opcode within Parrot's runloop
• that's fragile
• most exception problems now come from that
• this is the same problem as the continuation from different runloops problem
• longjmp doesn't know anything about the Parrot environment and doesn't restore it
• that's the corruption problem
• maybe the problem is how we handle Continuations

c:

• I think the problem is with how we handle Continuations

Patrick:

• there aren't many places where I do exception handling
• there may be a lot of handlers, but they're mostly in generated code
• not hand-written code
• might not need to update many places if exception handler as label goes away
• then the question is "How do I get to the context I had where the exception occurred?"

Allison:

• from my perspective, an exception handler is basically a block in the HLL
• the same way you'd use :outer with a block
• the outer is the enclosing sub in the HLL

Patrick:

• that's basically lexicals then
• anything I want to preserve into the exception handler, I have to stick into a lexical

Allison:

• yes
• or rely on the information you insert into the exception itself
• it can carry a payload

Patrick:

• but we're talking about exceptions I didn't write
• thrown from C
• I can't put stuff into the payload there

Allison:

• what information do you want?
• PIR information?

Patrick:

• and stuff in registers
• if I create a class, there could be a "Class already exists!" exception
• I branch around it
• if there's an error, handle it elsewhere in the subroutine, then continue on

Jerry:

• it's half-standard, when you write PIR
• half of the ops return NULL
• half of them throw exceptions

Allison:

• is that an idiom because Parrot's exception handling doesn't work?
• resume hasn't ever really worked so far
• if that worked, would that help?

c:

• that would solve some of them for me

Patrick:

• it would for me too
• but sometimes I want to do other things, and then resume
• I'd need to store stuff in lexicals to have access to them

Allison:

• the classic model is where you catch an exception, change a value, and then resume
• that's tricky to do with separate execution for handlers

Patrick:

• about half of my stuff is that
• mostly default handling
• testing to avoid the exception in the first place is a lot more work than trapping and fixing

Allison:

• and that makes exceptions less useful, as you want to take the hit only when an exceptional condition occurs

c:

• and there are race conditions too

Patrick:

• exception handlers as labels are useful, because you can call out to an external sub if you want

Allison:

• we'll have to do some work on Continuations to call them from within C without breaking things horribly

Patrick:

• before you go too far in looking at that, let me review where I use exceptions and how I use them to see what needs to change

Allison:

• the confusion is between the HLL and C
• they don't agree on how control flows

Larry:

• C is wrong
• the minute you put C and C in there, you're wrong

Allison:

• that's what I thought when I saw them
• I'll try to keep the ability to allow exception handlers at labels

Patrick:

• I'll look at how I handle C from a PCT perspective

Larry:

• C in Perl 6 allows you to go to the original scope from the exception handler
• the stack doesn't unwind until you tell it to
• it's not impossible... if you can find the right label to go to
• the C itself unwinds the stack

Patrick:

• what happens when you do C now?
• in trunk

Allison:

• it pushes a Continuation object onto the general stack

Patrick:

• when the exception occurs?

Allison:

• it pops it off the stack and invokes it
• when you C, you push the exception back on the stack

Patrick:

• anything that happens after you took that Continuation is still in effect when you get to that label

Allison:

• invoking a Continuation doesn't restore your entire environment
• only certain pieces of the context

Patrick:

• it doesn't change register values back

Allison:

• it's not pure
• in the sense of no side effects

Patrick:

• how does the new one work?

Allison:

• it doesn't use the general stack at all
• C adds a handler to the concurrency scheduler
• like event handlers
• when you invoke an exception, it searches through the concurrency scheduler's list of handlers
• invocation hasn't changed at all
• you'd store your exception information in the interpreter
• then pull that into an Exception PMC
• then throw that object
• we've removed that interpreter structure part
• superfluous
• Continuation invocation hasn't changed at all

Patrick:

• return exception returns a payload

Allison:

• you control that

Patrick:

• how about C and C?

Allison:

• are they all subs?

Patrick:

• PCT doesn't presume that the body of a C or C loop is a sub
• not all languages require that

Allison:

• I removed the idea of handler scope
• the old implementation wasn't very good
• my new idea is to store them in the context structure
• they only exist in your own context
• you attach those handlers to a block
• if you throw an exception within that block, the handler for that block catches it
• when you leave a context, the handler is no longer active

Patrick:

• that doesn't handle C
• have to stay in the loop
• it skips the rest of the block
• starts at the top again
• C exits the block

Allison:

• it hands control back to whatever controls the iteration?

Patrick:

• I need the C and C opcodes to work properly

Allison:

• the block catches it
• the handler says "resume after everything else"
• the ordinary control flow continues on the next iteration

Patrick:

• I need to think about it some more
• I could really use a way to push a new level into a lexpad
• some optimizations there

Will:

• Tcl has a C
• I'm not sure how it works under the hood
• it lets you do things with control flow

c:

• Lisp does too (CL?)
• let's take this to the list
• Bob Rogers has lots of practical experience doing this in Lisp
• good to get more feedback

The Perl 6 design team met by phone on 14 May 2008. Allison, Richard, Patrick, Jerry, Will, Nicholas, and chromatic attended.

Allison:

• working on the exceptions implementation for the concurrency branch
• also the Parrot foundation

Richard:

• trying to drum up funding for the Perl 6 paupers
• something's happened along these lines
• an anonymous member of the Perl community brokered a deal with a philanthropic business acquaintance
• I put together a pitch describing what we could do productively with a donation
• the money arrived yesterday
• half of it will go toward supporting Perl 6 development in a variety of ways
• mostly direct developer granting, like Patrick's grant
• the other half is for TPF use to improve organizational capabilities internally
• there's a lot more potential for funding
• orders of magnitude more
• that'll take us over the finish line
• though TPF is not a paragon of activity and organization
• we'll use it to improve to make us more credible to the rest of the world
• and potentially more developers
• want to go from a handful of grants like this to something larger
• I'll spend the next week or so making plans
• we'd like to start getting some of that out over the next few weeks

Patrick:

• outstanding work
• you'll go down in history for this

Jerry:

• will they money go out through the grant committee?

Richard:

• I don't think so
• this will be as directed by the board

Jerry:

• is there a PR plan for spreading this news?

Richard:

• I hadn't thought of that
• I'd like ideas for helping with that

Jerry:

• it's easy to post to web sites
• it might be good to give brian d foy a heads up for TPJ, too

Richard:

• I'm trying to do similar things for Perl 5
• this particular donor had a specific interest in Perl 6
• the half of this we're using to improve TPF will help Perl 5 as well

Nicholas:

• I hope the positive feedback will encourage other people to do the same for Perl 5
• presumably they're already using it and want to contribute back

Patrick:

• busy week here
• focusing specifically on Rakudo
• helping other people hack themselves
• added fat-arrow parsing, so now that works for passing named parameters
• added signature default values
• no more crazy MMD stuff to get that working
• worked on the test suite
• started at the beginning, Synopsis 2 cleanup
• making notes for others on things to do and not to do
• tried to close out old tickets, did about 18 Parrot-only tickets
• some updates to PGE to bring in S05 changes
• Tene added placeholder parameters to Rakudo
• surprised and pleased me to see that
• he found a problem with hashes versus arrays in PCT
• I did quite a bit of work to improve PCT code generation
• substantial improvements there
• generates much nicer code

c:

• there are some speed improvements there
• they're hard to measure, but they're there

Patrick:

• we're creating many fewer elements there
• especially in places like loops

c:

• we don't have many long-running benchmarks that would demonstrate that
• if we did, you'd really notice it

Patrick:

• it's been in the back of my mind since I started writing PCT
• it makes for nicer looking PIR code, if you like PIR code
• now I'm writing some journal posts to describe this to the wider community
• I don't know if anyone saw it, but there was a Twitter post that singled out TPF for its organization for Google's Summer of Code
• Jerry should get some credit for that for Parrot

Richard:

• I've talked to Eric Wilhelm a lot over the past few weeks
• he's doing a great job
• this has been his full-time job over the past few weeks
• that's why we're not always this organized
• very seldom do we have someone who can make such a thing a full-time job
• simple matters of organization often aren't

Jerry:

• worked with Allison on research for the Parrot Foundation
• she interviewed a lawyer to help set up the incorporation
• work has been lighter this week
• spent more time on Parrot and Perl 6
• haven't had any significant commits
• began to rewrite mod_parrot's configure system for flexibility and portability
• hope to get it compiling on Windows
• shouldn't be difficult
• Patrick and I discussed making HLL type mappings available in IMCC

Patrick:

• PIR doesn't seem to have a way of finding HLL type maps
• PCT needs to move something from a register into a PMC
• equivalent types, as far as the HLL is concerned
• there doesn't appear to be a way to do that
• without the HLL person repeating the mappings in PIR
• violates DRY

Jerry:

• I think I see a way of implementing that
• the major problem is that, internally, the PMC types are still type IDs
• not string names
• I might have a short-term fix
• the long term is to get rid of type IDs internally
• it's easy enough to expose that through interp_info.pasm or something

c:

• let me know if you need help

Patrick:

• also remember that class names are not always strings
• they are now, but the won't always be

Will:

• I'll try to unstick the type ID removal branch later this week
• mostly I've been doing queue management, which is not exciting

Patrick:

• I'm excited about seeing the number go down

Will:

• we've closed an enormous number of tickets in the last week
• yet more tickets keep coming in
• what's wrong with these users?

c:

• spent some time closing bugs and applying patches
• need to go through open patches and make sure they're all closed
• probably won't make much of the Bug Day
• backported yada yada yada operators to Perl 5, waiting for someone to apply them
• managed to speed up the Rakudo-building benchmark by about 40% yesterday
• hope to get onto the concurrency implementation soon

Nicholas:

• talked to Daniel Ruoso and stuck his response on the wiki
• otherwise busy with a potential move

Patrick:

• Jerry, are you still planning to reimplement abc to use the language shell tool?

Jerry:

• yes
• I plan to update the tool to include an operator precedence parser
• not sure if I should couple those changes

Patrick:

• I'm thinking of updating the tutorial
• kjs's tutorial is great
• I just want another one
• I don't want to step on your toes

Jerry:

• I plan to make my changes
• then apply the revised tool to abc
• I'll turn my attention to that next

Patrick:

• we should knock that out pretty quickly

Jerry:

• what's your plan for this release?

Nicholas:

• are you in a position to type things from this release to last release?
• or should I ask Patrick?
• you've sped up a lot of things

Patrick:

• Rakudo gets bigger over time
• there's quite a bit more code there than there was last month

c:

• there are a couple of ways to do that
• we could plot the ratio of time per feature over the release numbers
• that's not too difficult
• otherwise, we could take the current PGE/PCT/NQP/Rakudo and run it on this month's release versus last month's release
• or run last month's toolkit on this month's Parrot

Patrick:

• I can do that
• already thinking of ways to do that

Nicholas:

• Damian was talking about negative time last year
• you guys are doing it!

c:

• the big improvement is likely in the new garbage collector

Jerry:

• Andrew and Patrick should have a race

Patrick:

• longest token matching will help languages
• that's on my agenda for later in the summer though

c:

• remember that these are only Parrot-level improvements
• we haven't even started on PIR-level improvements
• because we can't profile it easily yet
• I suspect there are plenty of trivial 10 - 20% optimizations we'll find in hotspots there

The Perl 6 design team met by phone on 07 May 2008. Larry, Allison, Patrick, Will, Jerry, Nicholas, Jesse, and chromatic attended.

Allison:

• spent my time this week slicing and dicing the exceptions implementation
• replaced the old internals with the new system
• checked that in yesterday
• still a few failing tests in edge cases on the branch
• did more work on the Parrot Foundation

c:

• I own an acre of Mars, we could incorporate there

Allison:

• don't you own a cow in the Philippines?

c:

• yes, but that doesn't give me any governmental powers

Patrick:

• isn't that worth a lot?

c:

• the peso is improving against the dollar

Jesse:

• moving on...

Larry:

• clear bill of health from my medical reports
• hacking a lot against the standard grammar in my STD5 implementation
• lots of refactoring
• all of the various parameters that used to go through separately now go through as part of the Match object
• including the "fate" and whether we're peeking at the longest token set
• the longest token matcher works now
• I threw out my old mechanism for gathering Match objects
• it now creates the more-or-less correctly
• lots of grammar tweaks, as suggested by Mitchell Charity
• lots of refactoring of how logging works so that it doesn't always spew enormous quantities of information to the screen
• I can actually run the parser quite quickly now, for some definition of quickly that approximates 2000 characters per second
• matches symbols directly, rather than calling a rule, which is faster
• does the backoff now on longest token matching
• started refactoring the grammar on the assumption that I cna trust the longest token matcher
• no longer any nofat rule
• the longest token should match the fat arrow, if there is one
• started refactoring the quoting rules to parse as if they were sublanguages
• getting rid of extra rigamarole to recreate the other mechanism we already use for other languages
• working out the linkage for switching in and out of sublanguages
• how to get to the outer language from the inner language
• calling into pure Perl from closures in a regex
• or the host language if you're calling the regex from another language
• nailed down the available methods for Match objects in the specs
• giving a talk in Seattle on Friday at SPU
• flying to Japan on Saturday

Patrick:

• spent a lot of time teaching this past week
• cleared up now
• mostly I've answered questions on mailing lists and IRC
• I'm not always sure that I'm helpful, but I'm there
• yesterday I worked on trying to get a bunch of little small things here and there
• fixed up a few things in PCT internals
• today I'm bringing PGE up to date with some of the latest changes in S05
• these all help Rakudo and other languages in small ways
• trying to clean out my backlog and clean up a bunch of RT tickets
• I'll continue over the next couple of days
• and blogging about it as I go

Jerry:

• things are busy, mostly non-Parrot related stuff
• submitted a ticket that I hope Patrick can close today

Patrick:

• many languages depend on the old behavior, including Plumhead
• I'm not certain about some of them

Jerry:

• mostly otherwise answering questions on #parrot
• making sure that things are set up for the real work phase of GSoC
• making sure that students have their CLAs, if not commit bits
• astonished to see how much work Jonathan is getting done in just two funded days
• it's amazing to see how much a motivator money can be
• I'd like to see more of it, hint hint

c:

• working on closing as many open Parrot bugs as possible
• applying as many open patches as possible
• should be able to help on the concurrency branch soon
• otherwise preparing for the release
• going to check on received CLAs this week

Nicholas:

• found it curious that Perl 5.10 has the best state implementation of any language
• wanted to steal tests from another implementation
• had a discussion with Leon about SMOP
• there's no real description of how all of these implementations fit together
• Rakudo plus Parrot is a complete implementation
• SMOP and kp6 fit together nicely

Jesse:

• I started a wiki page on the Perl 6 wiki at Perl_6_Implementations

Patrick:

• I don't know that it says how things fit together

Jesse:

• I tried to encourage other people to contribute stuff
• didn't get much uptake

Nicholas:

• should we suggest to Daniel that he should help explain things?

Jesse:

• that's more likely to get people contributing to it

Will:

• there's definitely some confusion about it within the Grants Committee

Jerry:

• SMOP has the highest documentation-to-line-of-code ratio of any implementation

Patrick:

• it needs a good overview though

Nicholas:

• I'll ask Daniel to explain more
• especially its relationship to Parrot and Rakudo

Jerry:

• it sounds like it could be an alternate runcore for Perl 5 as well

Jesse:

• tried a few different things
• decided to write a test for Rakudo
• tried a simple arithmetic test pulled from Pugs
• found that Rakudo didn't implement a function specified in the S29 draft
• Patrick helped me write a couple of lines of code to implement it
• then discovered that fudge didn't support try blocks in a specific way
• Larry patched that
• then found that incrementing an undefined value didn't work in Rakudo
• that was the end of my day
• I still need to write up my findings
• how easy is it for someone without experience in Rakudo and its internals to pick things up and contribute something?
• more difficult than I thought it might be, but it's getting more doable
• it's important to understand how it might fail before trying to get people to do it
• then I started trying to play with MAD on the weekend
• found and fixed a bug in its XML
• refactored it such that you can run MAD's tests in the core if you add a copy of XML::Parser to the core
• it's not far enough yet, but it's a start

Nicholas:

• is it going to be difficult to restructure the Parrot foundation from 501(c)(3) to 501(c)(6)?

Allison:

• you can do pretty much the same thing
• sponsors are on the board in a c6
• they're only advisory in a c3
• the sponsors we've talked to are mostly only interested in getting regular status reports and the like

Jesse:

• is there any jumping around to transfer copyright to the new foundation?

Allison:

• we'll do a copyright assignment from the Perl Foundation to the Parrot Foundation
• all of the CLAs that went into the code up to the point of signover will be fine
• but we'll essentially copy the Perl Foundation CLA to a Parrot Foundation CLA

Will:

• do we need to contact committers who haven't signed a CLA?

Patrick:

• where does Rakudo fall?

Allison:

• still under the Perl Foundation
• it doesn't move at all

Will:

• do we want to split up the repository at that point?

Allison:

• eventually, we'll want to do so anyway
• it's not an urgent thing

Jerry:

• what would it take to version a Perl6Regex frontend to PGE?
• let grammars specify a version of the grammar

Patrick:

• I did that before by having a separate compiler
• you're talking about something a bit finer grained
• I don't want to do that
• as we get closer to 1.0, that'd be fine
• I already have enough to do keeping up with the latest versions

Will:

• I don't think we want to keep up old versions

Patrick:

• I don't mind sticking to our deprecation cycle
• I hadn't put the change from today into the deprecation list yet
• we'll get to it in a couple of weeks

Jerry:

• just trying to figure out how to push forward with changes to PGE without having to update every language in the repository

Patrick:

• freeze S05?
• not a great solution

Larry:

• I heard that

Patrick:

• the last few changes have been great
• I'm not really serious about that

Larry:

• some of them you even asked for

c:

• it's an advantage to have these languages in the repository
• we can update them
• but only if we can run the tests before and after and know that they pass

Will:

• we might consider removing languages with failing tests and no recent updates
• there are 17 grant proposals, some of them Perl 6-related
• please comment on the TPF blog
• it'll help

Jesse:

• blog.perlfoundation.org