Parrot: August 2008 Archives

The Perl 6 design team met by phone on 16 July 2008. Larry, Allison, Jerry, Will, Jesse, and chromatic attended.

Larry:

• switched my laptop to Ubuntu for various reasons, mostly work
• helped me get a runnable Pugs too
• haven't worked much on the standard grammar
• mostly thinking about refactoring the names
• thinking about various other questions people are asking
• just checked in a revamp of the closure semantics
• a little under the weather too

Allison:

• almost entirely absorbed by OSCON this week
• hacking a little bit on the three remaining test failures on the pdd25cx branch
• Patrick gave me some information on the PGE failures
• we'll work on that tonight
• hopefully will merge in the branch finally

Jerry:

• haven't done much this week
• will report for Patrick and Jonathan though
• Patrick's working on lexical issues
• he's much closer to understanding how Parrot can handle them
• he's also working with HLL to get it to work within PCT
• then he can make it work within high-level languages itself
• that'll let them use the same named classes as Parrot's base classes (for example, Integer) instead of funky workarounds
• Jonathan added enum support
• fixed a couple of segfaults
• something like 910 new spectests passing between Parrot 0.6.3 and 0.6.4
• that's a great number
• much of it came from the Summer of Code project for the Perl 6 test suite
• Adrian and Moritz have been a big help there

Patrick: (in a followup)

The primarily reason we want HLL support is to take advantage of HLL type mapping, so that str register-to-PMC conversion automatically gives me back a Perl 6 Str object instead of a Parrot String PMC. Same for int to Int, num to Num, and ResizablePMCArray to List conversions.

c:

• things are pretty slow
• traveling and haven't had much time
• still working with Andrew on the GC; he's making progress
• helping Allison; we're getting close
• will definitely start the string branch very soon
• maybe we'll merge that back in stages
• waiting for some guidance on the closure issue

Will:

• Parrot 0.6.4 came out on Tuesday, thanks to Bernhard
• no major new features, although the NEWS listing has lots of little updates
• hopefully will get the concurrency branch merged by the next release, to make it 0.7
• I hate Subversion
• I wonder when we're going to update RT on perl.org
• and that's it

Jesse:

• talked to Richard last week about other interesting funding sources and corporate support for Perl 6
• he's working on that
• general strategizing on that for pushing implementations forward
• sounds like the hackathon is fairly set post-OSCON?

Allison:

• yes

c:

• do we have a sense of prioritization of features, or is that up to implementors?

Jesse:

• it's been up to implementors so far
• every time I've asked, each one has strong feelings about how to prioritize

c:

• I mention it just because I believe that giving people something productive to experiment with sooner is very useful for marketing

Jesse:

• I've talked to Flavio about this, for example
• wondering if there are subsystems of features that could come before others

Jerry:

• what I really want in Rakudo right now is IO so I can use it for work
• it's limited
• you can't do sockets, basically
• that probably won't be fixed until it's fixed in Parrot
• that's Parrot blocking for Rakudo
• as far as other implementations go, there are early features that all implementations share
• I think Larry's said before that the Synopses deliberately don't look ahead too much
• if you do operators before you do objects, that's a way forward
• although parameterized roles are funded work
• and volunteer work

Larry:

• I do use those in my work

c:

• I don't have a sense of our priorities

Larry:

• Perl 6 will be very useful when it does its Perl 5 subset in a way that's better than Perl 5
• that said, there's a danger when Perl 6 gets used in anger by people who start to rely on transient bugs or workarounds for missing features
• there needs to be expectations management

Jerry:

• possibly the best way to do that is to say "Only use what passes in the spectest suite"

Jesse:

• unless there's a way to disable unstable features outside of devel checkouts

Larry:

• but what does it mean that a feature works?
• closure cloning might almost work, but there are some bugs
• we can put a large warning in the documentation that not all features are stable or final yet
• or run into the problem of installing Cabal features to run Pugs
• some packages are under development
• you're lucky if it all works
• it's easy to get an explosion of things that don't all work with each other
• I don't profess to have the right answer
• I don't think there is a right answer
• mostly I think the order of implementation will tend to iron itself out
• everyone has an itch to have the basic stuff working
• everyone has extra itches which may not correspond to other itches
• they're exercising parts of the design that other people aren't exercising
• I want convergence to happen

Jerry:

• provided you don't go crazy trying to keep it all in mind

Larry:

• there may be a need to name development subset versions
• maybe pre-6.0 isn't a sufficiently rich naming scheme
• at least it's an interesting problem

Jesse:

• it would be interesting to find a name for subsets of what will be Perl 6
• the trivial functional subset of Perl 6 that Audrey made in a week was a working subset of Perl 6

Larry:

• and it did junctions

Jesse:

• I've had multiple implementors prod me about how to name or label these things
• here's a thing we've targeted
• we can talk about it as an entity, even if it's not 6.0.0

Larry:

• there's a sense in which the true name is the list of all of the tests it passes

Jerry:

• Rakudo has a ROADMAP
• Patrick keeps it updated
• but he had some unanswered questions in that thread

Larry:

• there's a long list of things blocking from the Parrot end
• a lot of things are in perpetual redesign

Allison:

• I finished the last Parrot design document
• we need resources to implement them

Jerry:

• there were a lot of early prototypes that made it 60 - 80% of the way
• a lot of these features are waiting for a final thing
• it's no fun coding HLLs for a moving target
• they want to wait for the final version, rather than rewriting it several times

Jesse:

• the Rakudo ROADMAP is just a bullet list
• no subtasks, no estimates of effort
• a list in rough priority order
• it could use a couple of more levels of information

Jerry:

• we could make it a hackathon task

Jesse:

• I'll only be there for a couple of hours

Jerry:

• we could pull in some time earlier in the week
• why don't we call these chunks of features "traits"?
• that's an unused term

The Perl 6 design team met by phone on 09 July 2008. Larry, Allison, Jesse, Jerry, Patrick, and chromatic attended.

Larry:

• standard grammar parses itself 100%
• thinking about refactoring
• how do we distinguish the use of grammars from grammars used as the current language
• refactoring top level namespaces
• whether STD is part of that name
• meddling in everyone else's discussions
• upgraded the spec on transliteration
• in denial about all of the conferences coming up

Allison:

• absorbed by OSCON and travel for the next couple of weeks
• after that, things will pick up
• working on one (hopefully) final patch to the concurrency branch
• I know it'll resolve one failing test
• hope it'll resolve several PGE failures on the branch
• I'm an hour away from committing that (ideal minutes, but probably an actual hour)
• we're talking about having a hackathon the Saturday after OSCON

Jesse:

• is there a venue?

Allison:

• nothing set up yet, but we do have a default

Jesse:

• how many people?

Allison:

• probably about ten
• Portland State University worked out pretty well for us
• I'll look for contact information

Patrick:

• last week, we passed 1365 tests
• now we're passing 1677, 312 more
• from the last Friday of June to the first Friday of July, we had an increase of 500 tests
• almost 50% more than we were

Jesse:

• how many are newly ported tests and how many used to fail?

Patrick:

• probably an equal mix of both
• a big part of that is getting implicit lexicals working, $_, $!, and $/
• implicit method calls too are working now
• lots of refactoring, especially for builtins
• they now seem to work better in the Any class
• removed a lot of inline PIR code and stuff like that
• spent an hour on the phone with the Houston Perl Mongers last night
• they want to work on the test suite as part of their meetings for a while
• I walked them through building Rakudo, checking out the Pugs repo and test suite, explaining how we do testing
• they plan to have meetings where someone gives a topic, presents how they think it should work, and then they review or create tests for that feature
• they'll either move tests from Pugs into spectests, or add tests to spectests
• I'll try to write that into an article for use Perl or other places
• a step by step guide for contributing tests
• haven't started writing my "This Week in Perl 6 Articles" yet
• should have one by this time next week

Jerry:

• haven't been able to produce much code
• participating in code reviews and design discussions
• this is the mid-term evaluation period for Google Summer of Code
• all of the mentors and students have submitted their mid-term evaluations
• it's all gone smoothly
• all three are going ahead
• the Parrot Foundation is in process of taking over the NLNet grant from TPF
• working with the OSU open source labs to host the website
• expect more on that soon

Allison:

• because we need SSL certificates, they were going to set us up with our own virtual machine
• we'll need to purchase the certificate

Jerry:

• and finalize the purchase of parrot.org

c:

• applied some patches, fixed a couple of bugs
• closed some tickets, but a lot more came in
• worked with Andrew some on fixing bugs in his GC
• getting closer to the point of merge
• approved his milestone, setting some priorities for the second half of the work
• probably won't have much time to work in the next week
• thinking about Closures, not sure Parrot's approach is entirely right

Patrick:

• did you see my note about the PGE bugs in the branch?
• I ran PGE in the concurrency branch yesterday, and get an odd result
• if you run it by using prove, it gives errors
• if you run the PIR directly, it works just fine

c:

• I don't think Parrot::Test adds any flags when it invokes Parrot
• but t/harness does

Patrick:

• it's in the operator precedence parser
• you can specify a particular stop token
• nothing uses that anywhere anymore, but the test tests it
• I don't think anything that uses it

Allison:

• does it throw an exception?

Patrick:

• it just stops parsing at that point
• the caller decides what to do
• it's not detecting that token when it's parsing
• I pass a string with a stop token in the middle
• it's supposed to parse only the first part
• but it doesn't stop at the stop token
• it's just a simple comparison of some sort
• seems kinda weird

Allison:

• maybe it is detecting it, but isn't working right
• but if you're not using exceptions, there's no old exception handler lying around
• so it's not that

Patrick:

• I gave instructions on how to reproduce in my message

Allison:

• my current patch probably won't fix that
• it sounds unrelated

Patrick:

• I might put some tracing in and run it through the harness
• figure out why it's failing to detect that stop token
• also a question for Larry
• do we have an official leaning on method fallback?
• are they in or out?

Larry:

• assume they're out for now

Patrick:

• I'm happy
• I don't have to rewrite my dispatcher
• can I start telling people to organize the test suite that way?

Larry:

• sure

Patrick:

• for the eval built-in, can someone take String.eval and have it do what you expect?

Larry:

• yes

Patrick:

• a method that's exported
• great!

Jerry:

• STD.pm parses itself
• that's fantastic
• is the LTM implementable now in Rakudo or elsewhere?

Larry:

• essentially

Jerry:

• is anything blocking that?

Patrick:

• just time
• but I plan to develop that under the grant
• I think it'll be a significant chunk of code and research
• don't want to do a bunch of PGE refactoring until the concurrency branch is back in trunk though

Jerry:

• I want to get that merged as soon as possible
• it won't make a lot of progress until we do
• hope we can get it before OSCON, even though Allison will be busy

Allison:

• there are only three failing tests

Jerry:

• but HLLs don't work

Patrick:

• I don't want to go into OSCON without languages working on trunk

Larry:

• maybe a reverse merge?

Allison:

• we do that regularly

c:

• I just don't want to block everyone from doing anything until the code gets fixed

Patrick:

• just about everyone who can fix it is on this call

Allison:

• we're almost to the point where it's time for HLL implementors to work on it

c:

• seems like we can pick the top three HLLs that need to work (Rakudo, Pheme, Punie)
• make them pass
• give the other HLL implementors a week to fix their stuff
• then do the merge

Patrick:

• just don't want to show off all of this nice stuff at OSCON, then say "Don't download trunk, it doesn't work"
• but show off the new stuff in Rakudo that's not in the new release
• a week makes a big difference, the way things are moving

c:

• that could be 500 tests

Patrick:

• thanks to Larry for jumping in on the design discussions
• hope you're not frustrated

Larry:

• it's all part of the job
• I'd rather see progress than vice versa

The Perl 6 design team met by phone on 02 July 2008. Allison, Will, Jerry, Patrick, and chromatic attended.

Allison:

• still working on the pdd25cx branch
• taking way too long — five remaining test files
• I'm going to have zero tolerance for failing tests, even on a branch
• I broke my natural tendency for that and it's biting me
• I'll be traveling a lot for the next couple of weeks
• hope to get one or two tests passing tomorrow
• two of the failures are PGE, which probably means that no language built on PGE works now, depending on where the test failures apply

Will:

• Tcl still fails a ton of tests on the branch now
• that's not far off the mark

Jerry:

• do you know the root causes?

Allison:

• not in PGE
• but the one I'm working on now is a fundamental problem
• exception handlers used to be scoped
• I shifted them over to global in the first stage
• now I'm shifting them over to the context, so they'll behave like the old system
• the old system used a global stack, which worked badly with CPS
• I hope that this change will fix a bunch of the PGE tests

Jerry:

• Summer of Code is progressing nicely
• we're coming up on midterms now
• everyone seems right on track for the Perl 6 and Parrot projects
• the students are all engaged in their work and the community
• I have high hopes that they'll continue to be around after the summer
• Patrick and I need to coordinate our OSCON talk
• have only had time for administration and question answering
• don't know what my availability will be after next week, but I'll do my best

Patrick:

• this year's Summer of Code is far more productive than previous years, from the Perl 6 and Parrot perspective
• the work that they've done will stick around
• it's really, really helping out
• Rakudo now passes 1365 tests out of the spec repository — 422 over last week
• increase of 75 over this morning
• probably another 50 or so, after the lexical issue is now resolved
• I can fix $_ and implicit method calls
• I think that there are a lot of tests that use those; keep noticing some that aren't passing
• it's been a banner week for adding new tests passes
• some of it is improvements in Rakudo
• moritz and auzon are also doing a great job in refactoring the tests, and making sure Rakudo passes them
• every so often they find a Rakudo bug, we patch it, and we have more passing tests
• Jonathan and chromatic managed to fix lexical handling over the past week
• that removes a big blocker for me
• I cleaned up part of PCT
• about to clean up part of Rakudo
• we should get better code generation out of that
• plan to refactor the builtins to become part of the Any class
• will fix $_, $!, and $/ to work correctly
• will refactor parameter passing and handling
• wrote my third progress report for the Mozilla grant
• the final report will come out around the time of OSCON

c:

• fixed some bugs
• trying to remove as many blockers for Rakudo and other Parrot languages
• continuing to work with Andrew to make sure his new GC branch compiles and runs and passes all tests

Jerry:

• he's at 700 failures out of 7700 now right?

c:

• it didn't even compile a few nights ago
• so he's making good progress

Will:

• trying to rip out old stuff from Parrot
• added a :deprecated flag for ops
• if you run Parrot with warnings, it'll warn in almost every program
• started a branch to remove built-in method handling
• magical non-ops dispatch to class methods on PMCs
• we're trying to get rid of the old object system
• the only one we're really using right now is say anyway
• we don't have to worry about having too many opcodes right now anyway
• removed a lot of custom code for our Perl::Critic configuration
• increased our dependencies, decreased custom code
• fixed a bug for subclassing Float
• we have a whole class of bugs where when you subclass a PMC, the PMC assumes you have the same C type
• we need someone to do a review on all of that
• the RT #48014 is a good place to comment on how to migrate from using the PMC union to PMC attributes

Allison:

• are we doing any hackathoning after OSCON?

Patrick:

• I have Saturday open, so we can do that

Jerry:

• I can do that too

Allison:

• Larry left Saturday and Sunday open

Patrick:

• I can move my flight; I have a place to stay on Saturday night too

Allison:

• if it's just you and Larry, that might be valuable too
• that's five or six of us
• where's a good location?

Patrick:

• it depends on who's coming
• will Damian make it?
• there could easy be six or seven of us

Allison:

• I'll look into spaces
• let's say Saturday for sure, but Sunday is possible

c:

• and Friday afternoon and evening

Jerry:

• I'm meeting with Hank at Microsoft's open source labs
• he's offered some resources, right?

Allison:

• smoke testing
• some Windows licenses

Jerry:

• now that we're very close to having Smolder work
• (today or tomorrow)
• just requires an upgrade to TAP::Harness 3
• we could make this happen
• I'll basically follow up and make things happen

Allison:

• they want something automatable, so they don't have to think about it
• it'll update once a week or whenever, run the tests, and submit a report

Given that most of July and August for me was spent attending conferences, travel, vacation, and periods of limited network connectivity, it's been a while since I've been able to update Rakudo progress. So, this is a very brief update, to be followed by a longer post in a day or so.

First, we continue to make progress on passing more spectests. As of this writing Rakudo is passing 2278 spectests. A graph of our progress is available from http://www.pmichaud.com/perl6/rakudo-tests-2008-08-25.png. Much of the credit for passing tests over the past few weeks goes to Jonathan Worthington, Moritz Lenz, and Carl Mäsak (again, apologies if I've overlooked anyone). It's really good to see progress coming from so many sources.

OSCON and YAPC::EU were both excellent conferences, my compliments to the organizers of each of those. In particular, Jonathan, Allison, and I were able to use our time together at YAPC::EU to discuss and map out many of the outstanding issues for Parrot and Rakudo, which we can now begin documenting and implementing over the next few weeks and months. These include things like code initialization, handling Perl 6 parameter passing and signatures, MMD, strings, and many other issues.

One of the outcomes of this was that early last week I finally got the code in place to enable pre-compiled Perl 6 modules to function properly. At the moment this has a quite dramatic effect on running the spectest regression suite, because we aren't having to parse and recompile Test.pm on each test file execution (and parsing is still our biggest bottleneck). So, running the regression suite dropped from twelve minutes to under four minutes on my system, which isn't quite so interminable.

Of course, the next step will be to enable Rakudo to compiler Perl 6 programs into standalone PIR or PBC files that can automatically load the Perl 6 runtime. I expect to accomplish this sometime this week -- at present we need some refactors to the signature generation code that is blocking this from happening.

We're also working on enabling parts of the standard runtime library ("Prelude") to be written in Perl 6 and precompiled by Rakudo, instead of having it all written in PIR. As part of this we may implement a Perl 6-with-inline-PIR capability to help with the builtins, to make it easier to attach MMD signatures to the builtin functions and make sure they're exported properly.

We also have more of the interface for loading external modules (written in PIR or otherwise) specified, and will be working on that over the next couple of weeks.

At the YAPC::EU hackathon, Jesse Vincent and I also spent some time updating the Rakudo ROADMAP document. This newer version of the ROADMAP identifies some of the specific components that are left to be developed, along with estimates of their complexity and dependencies. If you look at the document you'll see some notations about how long it will take to develop each feature; these estimates are all in "idealized programmer units". An "idealized programmer unit" here assumes that the people involved have no interruptions or distractions, has all of the needed prerequisites in place, is mentally charged and ready for programming, doesn't have to wait to coordinate questions or answers with others, etc. As such, the times given should be taken only as relative estimates of task difficulty, and not how long a particular task will take in real-world time units.

The major subsystem redesign coming up for Rakudo will be changes to the parser and grammar engine to support protoregexes and longest token matching. This will enable us to support even more of the STD.pm "standard grammar". I expect much of this work to take place over the next four calendar months, as many elements are likely to require some intensive and sustained design and development effort (while trying to maintain progress in other areas). More on this as it progresses.

Pm