Parrot: September 2008 Archives

The Perl 6 design team met by phone on 20 August 2008. Larry, Allison, Patrick, Jerry, and Jesse attended.

Larry:

• decommuting from Copenhagen
• couple of tiring days wandering city
• mostly hacking on STD.pm grammar to speed it up a bit
• mostly propagating ratchet optimization around
• compiles itself in under 90 secs
• it's an improvement
• also working on adding rudimentary symbol table checking
• can figure out where it's misparsing things as function calls
• getting ready for trip to Japan next week
• lightweight languages conference, for some definition of "lightweight"

Patrick:

• returning home from Copenhagen, which was quite productive
• on the plane ride I implemented pieces for precompiled modules in Rakudo
• finished that up last night, so now Rakudo can use precompiled modules
• spectest_regression on my box went from 12 minutes to under 4 minutes
• it also means that use statement can start doing load_bytecode things
• today been working on lazy lists with jonathan
• much of my time in the next couple of days will be writing reports and blog posts, catching up on events at OSCON and YAPC::EU
• also, at YAPC::EU hackathon Jesse and I put in some refactoring on Rakudo's ROADMAP, identifying task dependencies and some relative effort required

Jerry:

• GSOC update; mentors reviewing and submitting final reports
• looks very good that all three are completing
• very successful SOC
• we have good NCI stubs for Parrot, no longer need to write function signatures for things in C header files
• have 700 new tests added by Adrian for Perl 6 spectests
• have mostly complete GC implementation (some debugging needed)

Jesse:

• the recent blog post sounded a little "unhappy" with outcome

Jerry:

• yes, that the student's self-perspective
• however, it's design-wise complete, just not debugged fully

Allison:

• we didn't expect total rewrite of GC system
• we can do the GC integration with Parrot
• he did what we needed him to do, indeed he went much farther than I expected

Patrick:

• chromatic also mentioned that things are farther along than intimated
• just need a bit more debugging before we can pull it back into trunk
• didn't want to potentially block others' progress

Jerry:

• trying to catch up on work done at YAPC::EU
• much didn't happen online
• missed out on some good conversations
• today discussed exportation in Parrot on IRC with Patrick

Jesse:

• getting way too little sleep at YAPC
• very nice conversations with Patrick
• developed ROADMAP, put estimates of relative effort on them
• there's a lot less left to do than I would've estimated
• I also talked to Jonathan about what he'll be up to after his current grants expire

Allison:

• reverse commute from Copenhagen
• diving right into mmd branch, have tasklist
• Andrew and smash very helpful
• finished parsing of multi declarations
• build multi-dispatch table from C declarations
• then deprecate a bunch of C code to get to a single mmd impl
• roll that branch back into trunk
• should finish MMD milestone this week or next
• then it's onto IO implementation

Jesse:

• Perl 6 IO specification was flagged in Saturday's discussions
• Where does that belong?

Larry:

• I've looked at it a few times, it's certainly not complete
• "It's just a library issue!"
• it needs some TLC

Jesse:

• Are there others worth harassing for proto-TLC?

Larry:

• where IO interacts with concurrency I tend to sweep under the carpet
• I tend to not stress about it
• let's do one step better than P5 and not worry about it

Patrick:

• I also think that as long as we can soon provide an implementation that can do basic file I/O and sockets, then people can explore and play in the design space

Larry:

• as long as it's properly OO to begin with, there's lots of room to play

Allison:

• Parrot's IO model is becoming much more OO, yes

Jesse:

• could Rakudo take care of providing an OO interface to Parrot's IO?

Allison:

• independent of Rakudo, Parrot IO needs to be OO
• the old design seems to be "reinventing multiple systems to do the same thing"
• also wanted to mention volunteers working on test.parrot.org
• moving pages over to new parrot.org domain
• as soon as pages are moved over, we just need security certificate

The Perl 6 design team met by phone on 06 August 2008. Larry, Jerry, Nicholas, Jesse, and chromatic attended.

c:

• started the release management expectation for post 1.0
• want to get that nailed down before the end of the year
• applying patches, fixing bugs
• more small projects, the better
• helped Allison get the concurrency branch merged, looks fairly good

Jerry:

• agree with that timeline for getting documents sane on the release process
• lots of traveling for several Parrot and Rakudo hackers
• work has slowed down because of that
• the concurrency branch merge was pretty smooth
• able to dedicate more time to Parrot and Rakudo lately
• reviewing Kevin Tew's GSoC project as it nears completion
• running into a miniparrot build problem
• trying to help him debug that
• taken over mentoring for Adrian while Moritz is on vacation
• he's been pretty quiet this week

Larry:

• the standard grammar parses 99+% of t/
• four of the six failing tests are the fact that it does not install new grammar rules for user-defined operators
• one of the tests fails because the author of the test thought that Perl 6 as should read your mind as to where to apply adverbs
• straightening out some of the parsing snafus
• we need to be able to distinguish between types and variables all over the place
• now :: really isn't a sigil
• it gets in the way all over again
• special symbols like ::?CLASS are specially parsed at this point
• responding to questions on p6l about exception handling with defined-or and MMD semantics
• mostly just hacking on the standard grammar and the tests
• always have to decide whether the test or the grammar is wrong, when they conflict
• thankfully it's often the tests
• going to Copenhagen next week

Nicholas:

• part of the problem of Perl 5 and the CPAN gets solved with Perl 6
• which allows different versions of modules by different authors

Jesse:

• had several chats with Audrey, who's hacking on Perl 6 again
• she's taking Pugs apart into separate useful packages of Haskell code for Hackage
• it's a Voltron-style development method for Pugs
• the current version passes about the same number of tests as the version of two years ago
• it now passes them in 15 minutes, instead of two hours
• a dramatic performance improvement already

Nicholas:

• why?

Jesse:

• some internals changes
• she's documented some of them
• she's waiting for the 6.10 version of GHC, which adds built-in DSL support
• it'll be a lot easier to port the standard grammar for running in Pugs atop GHC
• it's easier to drop pure Perl 6 implementations of Perl 6 on top of Pugs and get them to work then
• now a lot of cleanup and stabilization is going on
• she wants a lot of people hacking on Rakudo, v6.pm, and the tests
• also Haskell hackers
• Flavio's back to hacking, but we haven't caught up
• there's a lot more activity now than three weeks ago
• I'm presuming there's no call next week, with so many people at YAPC::EU

Nicholas:

• applied a patch to get parallel tests working in the Perl 5 core
• running full tests in parallel is only slightly faster than serial

Jesse:

• use -j 9 on a dual-core machine with the new Test::Harness
• --fork also works well
• --save and --slow front-loads the slowest tests, which causes things to run faster

Nicholas:

• Andy Armstrong says this is moderately brittle
• I think he pays more attention to issues from p5p at the moment than Parrot

Last year I received a Perl 6 development grant from the Mozilla Foundation and The Perl Foundation. Below is a copy of the final report I've submitted to close out the grant. It's also available in PDF if anyone wants a more printable version.

My thanks to everyone who helped me with this grant, and I'm looking forward to the next one.

[2008-09-11 update: the original version of this report mis-attributed the Squaak tutorial to Kevin Tew; the corrected author is Klaas-Jan Stol (kjs). My sincere apologies for the mistake, it is now corrected below.]

----------

Perl 6 Development Grant
Final Report

Patrick R. Michaud
September 10, 2008

INTRODUCTION

This is the final report for the Perl 6 and Parrot development grant provided by the Mozilla Foundation and The Perl Foundation. This report summarizes the work performed under the grant, what has been accomplished, and where things are headed from here. As this report illustrates, we have achieved all of the goals and outcomes intended for this grant.

REPORT

The primary focus of the project was to get Perl 6 on Parrot development "over the hump" and acquire a critical mass of infrastructure, developers, and tools from which sustained further development of Perl 6 can take place. The project proposal identified five specific expected outcomes:

1. A working Perl 6 on Parrot implementation that supports commonly used Perl constructs and operators
2. Review and improvements to the Perl 6 language test suite
3. A substantially complete Parrot Compiler Toolkit, with documentation
4. Ongoing, active development efforts for other languages based on the Parrot Compiler Toolkit
5. An increased number of participants in Perl 6 and Parrot design, implementation, and testing

1. A working Perl 6 on Parrot implementation

We now have a substantial Perl 6 implementation on Parrot. At the beginning of this project, we had a rudimentary Perl 6 compiler that could handle some of the basic features and syntax of Perl. However, some key features such as arrays and hashes were only marginally implemented, and most of the translation components of the compiler were written in Parrot's intermediate assembly language (PIR).

Today the Perl 6 on Parrot compiler has been substantially rewritten using the Parrot Compiler Toolkit (described below), and is now known as "Rakudo Perl" or "Rakudo" [1]. Rakudo currently supports arrays, hashes, classes, objects, inheritance, roles, enumeration types, subset types, role composition, multimethod dispatch, type checking, basic I/O, named regular expressions, grammars, optional parameters, named parameters, slurpy parameters, closures, smart match, junctions, and many other features expected from Perl 6. Rakudo has progressed to the point that others are now using Rakudo as an Apache handler ("mod_perl6") [2], a wiki engine ("November") [3], and recently to build applications on Xlib [4]. The Perl Foundation also recently awarded a grant to Vadim Konovalov to implement a Tk GUI interface for Rakudo [5].

During the project there have been many contributors to the development of Rakudo Perl; some of the major contributors include chromatic, Vasily Chekalkin (bacek), Jerry Gay, Jeff Horwitz, Moritz Lenz, Carl Mäsak, Cory Spencer, Stephen Weeks (Tene), and Jonathan Worthington. Jonathan Worthington's efforts deserve special mention: he is primarily responsible for the bulk of the work on classes and types in Rakudo Perl, and much of his work was supported by a grant from Vienna.pm [6].

2. Review and improvements to the Perl 6 test suite

At the beginning of this project, the test suite that existed for Perl 6 had been implemented as part of the Pugs effort and contained approximately 16,000 tests. However, in many cases the tests were out of date with respect to the current Perl 6 language specification, and we needed a way for multiple independent Perl 6 implementations to be able to easily make use of the test suite.

In December 2007 I proposed a structure for reorganizing the test suite [7]; Jerry Gay and Larry Wall then extended this proposal and developed tools to make it easier to share the tests among multiple implementations. In May 2008 Moritz Lenz refactored Rakudo's test harness to provide a "make spectest_regression" target -- since then this has become our primary measure of Rakudo progress [8]. Moritz Lenz also developed a tool to display the progress in graphical form:

Adrian Kreher received a Google Summer of Code grant (with Moritz Lenz and Jerry Gay as mentors) to continue the test suite refactoring [9]. As of early September 2008, the official, refactored test suite contains a little over 8,000 tests, or approximately half the size of the original Pugs test suite. Furthermore, the official suite includes many new tests for object-oriented and typing features of Perl 6 that weren't present in the original test suite. As the graph above indicates, Rakudo is currently passing over 3,200 of the tests in the official suite, and work is continuing on refactoring the suite and increasing Rakudo's pass rate.

3. A substantially complete Parrot Compiler Toolkit, with documentation

As mentioned previously, the compilers that existed for Parrot at the beginning of the project (such as Perl 6) were primarily written in Parrot's intermediate assembly language. This made working on the compilers less accessible to new developers, as well as increasing the time needed to build a compiler. Therefore, the first couple of months of this project were spent on developing the Parrot Compiler Toolkit (PCT) and a simple Parrot language called Not Quite Perl (NQP). PCT provides an abstract syntax tree representation and code generator for Parrot languages; NQP enables compiler writers to easily create compilers and builtin functions for Parrot using a simplified Perl 6 syntax.

Once PCT and NQP were substantially complete, we were then able to convert Perl 6 (now "Rakudo Perl") and many other Parrot compilers to use the new toolkit. This went surprisingly quickly -- most of the existing compilers were converted within just a couple of weeks. In addition, a few new compilers and languages arrived on the scene: Will Coleda and Simon Cozens quickly created an implementation of LOLCODE [10], and Klaas-Jan Stol created a language called "Squaak" as a demonstration and tutorial for the toolkit [11]. A couple of quotes from the period give a sense of how these tools opened up Parrot development to others:

"The real fun, though, has been digging into perl6, the Parrot Perl 6 implementation. Recently, Patrick Michaud has been doing some incredible work building NQP (Not Quite Perl 6), a bootstrapping language for implementing Perl 6, and extensively refactoring the existing Perl 6 on Parrot compiler to fit with it. I'm still very much getting my head wrapped around the whole thing, but it's been easy enough to start digging into and implementing and fixing a few things."

- Jonathan Worthington, December 2007 [12]

"It's really, really true. Parrot lets you implement your own languages using Perl 6 rules for the grammar and Perl 6 for the compiler."

- Simon Cozens, January 2008 [13]

Both PCT and NQP stabilized early in the project; recent changes have been primarily to optimize existing features or make other minor improvements. We expect these tools to continue to evolve as needed to support compiler development; however, given the wide variety of languages being implemented using the toolkit surprisingly few changes have been needed. Primary documentation for the toolkit consists of a Parrot Design Document (PDD26) for the abstract syntax tree representation [14], the Squaak tutorial [11], and numerous example languages in the Parrot repository. More detailed documentation and examples for the toolkit are expected to be developed over the coming months.

4. Ongoing, active development efforts for other languages based on the Parrot Compiler Toolkit

Currently there is active development on at least three languages for Parrot; these include Perl 6 (Rakudo Perl), PHP (Pipp), and Ruby (Cardinal). In addition, there is ongoing but less active development on implementations of Python (Pynie) and Smalltalk (ChitChat). All of these are based on the tools from the Parrot Compiler Toolkit. The Perl 6 and PHP implementations are usable from mod_parrot [2], and our next steps are to improve the toolkit and library conventions to support loading multiple languages simultaneously in Parrot.

5. An increased number of participants in Perl 6 and Parrot design, implementation, and testing

There can be little question that momentum for Perl 6 and Parrot development continues to grow, and the work supported by this grant has been a major catalyst for that growth. For the twelve months prior to September 2007 the Parrot subversion repository had a total of 6,634 commits; in the subsequent twelve months Parrot had 9,686 commits -- a year-over-year increase of 46% in the commit rate.

At the beginning of this project there were perhaps three or four active contributors to the Perl 6 on Parrot compiler (i.e., Rakudo Perl); over the course of the past year that number has increased to at least ten active contributors and at least as many more occasional contributors to Rakudo Perl. Parrot and the Parrot Compiler Toolkit have also garnered new contributors, including several new committers. There are also new teams of developers working on applications based on Rakudo Perl and Parrot, such as the November wiki engine and the Rakudo/Tk GUI interface.

Lastly, recent improvements in the Rakudo compiler architecture will allow much of the Perl 6 runtime library currently written in PIR to be rewritten substantially in Perl 6. This will enable even more new contributors to participate in Perl 6 development.

PROJECT EVALUATION

The project proposal identified five criteria by which the success of this project could be measured:

• Ability to write and test Perl 6 programs and language features
• Passing rate for Perl 6 language test suite
• Improved coverage and accuracy of the Perl 6 language test suite
• Increased number of participants in Perl 6 and Parrot development
• Active development of at least two other languages using Parrot compiler tools

This report demonstrates the success of this project along all of these measures. Perl 6 on Parrot ("Rakudo") is being used to write and test Perl 6 programs and language features, and has become a strong driver for improving the coverage and accuracy of the test suite (and indeed, of the Perl 6 language specification itself). Continual reporting and publication on Perl 6 and Parrot activities continues to attract increased interest and participation from the wider community.

Moreover, this project helped jump-start even larger fund-raising efforts. In May 2008 the Perl Foundation received a $200,000 philanthropic donation from Ian Hague; roughly half of this donation is intended to continue the Perl 6 development efforts that have been part of this project [15]. And, as mentioned earlier, Jonathan Worthington and others are receiving grants for continued work on Rakudo Perl and Parrot [6,8,16].

CONCLUSION AND FUTURE WORK

The funding provided by the Mozilla Foundation and The Perl Foundation has indeed enabled us to get Perl 6 on Parrot development "over the hump" in development. We now have a robust platform for higher-level language development on Parrot, along with an active and growing development and support community (which is the hallmark of any successful open source project). All of the desired outcomes of this project have been realized.

Our next steps will be to continue to extend and build upon the work of this project -- increasing Rakudo's coverage of the Perl 6 language and bringing all of our efforts much closer to production releases. In fact, we recently developed a "road map" for Rakudo Perl that identifies the major steps to be taken in the upcoming months and assists in coordinating the remaining development activities [17].

Lastly, I want to express my sincere appreciation to the many people who contribute time and energy to Perl 6 and Parrot, and to give special thanks to the people of the Mozilla Foundation and The Perl Foundation for their ongoing support and enthusiasm for this project. It is a great honor to work and correspond with such a terrific and professional group of individuals.

REFERENCES

1. P. Michaud (January 16, 2008), "The compiler formerly known as 'perl6'", http://use.perl.org/~pmichaud/journal/35400
2. J. Horwitz, "Mod_parrot website", http://www.smashing.org/mod_parrot/
3. C. Mäsak, "Announcing November, a wiki in Perl 6", http://use.perl.org/~masak/journal/37212
4. http://svn.perl.org/parrot/trunk/examples/nci/xlibtest.p6
5. Alberto Sim&otild;es (August 30, 2008), "2008Q3 Grants Results", http://news.perlfoundation.org/2008/08/2008q3_grants_results.html
6. "Vienna.pm funds Jonathan Worthington to work on Ra[kudo]" (April 23, 2008), http://use.perl.org/article.pl?sid=08/04/23/2314234
7. P. Michaud (December 20, 2007), "Proposal: refactor the test suite according to synopsis, http://groups.google.com/group/perl.perl6.compiler/msg/ed748490f030da2f
8. P. Michaud (June 16, 2008), "Rakudo test suite progress", http://use.perl.org/~pmichaud/journal/36695
9. A. Kreher, "Auzon's Blog: gsoc2008", http://auzon.blogspot.com/search/label/gsoc2008
10. "I HAZ A PARROT" (January 3, 2008), http://lolcode.com/news/i-haz-a-parrot
11. K. Stol (March 9, 2008), "PCT Tutorial Episode 1: Introduction", http://www.parrotblog.org/2008/03/targeting-parrot-vm.html
12. J. Worthington, "Chipping away at perl6", http://www.jnthn.net/cgi-bin/blog_read.pl?id=589
13. S. Cozens (January 3, 2008), "Parrot is really quite wonderful", http://blog.simon-cozens.org/post/view/132
14. "Parrot Abstract Syntax Tree (PDD 26)", http://www.parrotcode.org/docs/pdd/pdd26_ast.html
15. R. Dice (May 16, 2008), "TPF receives large donation in support of Perl 6 development", http://news.perlfoundation.org/2008/05/tpf_receives_large_donation_in.html
16. J. Worthington (August 5, 2008), "Multiple Dispatch Design Work", http://use.perl.org/~JonathanWorthington/journal/37101
17. "Perl 6 Development Roadmap", http://svn.perl.org/parrot/trunk/languages/perl6/ROADMAP

The Perl 6 design team met by phone on 30 July 2008. Allison, Jerry, Will, Jesse, and chromatic attended.

Allison:

• OSCON was a success
• now I'm completely free from all responsibilities until at least December
• launched the PIR PDD
• now working on the last few failing language tests before I merge back the concurrency branch
• also spent some time talking to PhD advisors at Cambridge this week
• might start fall 2009, if I can get funding and such
• leaving for London on Saturday
• speaking at the BBC on Tuesday
• may miss next week's meetings

c:

• was very busy with OSCON
• trying to apply a few patches
• trying to get other people to the place where they can help with that
• mostly I need tasks I can do in 30 minutes when I can grab an hour or two
• fixed some small stuff at the hackathon

Allison:

• I need to split up the MMD PDD into bite-sized tasks

Jerry:

• OSCON was fabulous
• our talk was successful
• good feedback and enthusiasm
• the lighting talks with Jeff were successful
• most of my hacking was on mod_parrot
• trying to refactor its configure system into something more robust and portable
• it's just about working in the same state it was before
• hope to reach that point this afternoon
• met with some folks at Microsoft's open source lab on Monday
• promising beginning to a relationship between them and the Parrot Foundation
• the hackathon was productive as well

Jesse:

• I talked to some Sun people about what makes CPAN tick

Will:

• mostly absent the past couple of weeks
• trying to resurrect Tcl, which was untouched for some time
• making progress there on spec tests
• identifying what's left to do

c:

• what's our roadmap look like for the rest of the year?

Jesse:

• I asked Patrick about that
• he said there's a good chance to get that from the hackathon
• if not, he said I could get his attention at YAPC and get his milestones to break down soon

Jerry:

• he spoke on IRC about that today
• he wants to break it down into better granularity and put dependencies between tasks
• have a feeling it'll happen at YAPC::EU
• Jonathan is keen to get together with Allison and Patrick there to discuss related designs

c:

• I've noticed projects in Parrot like Reini's attempt to get installation and loading sorted out
• seems like time to get those sorted out

Jerry:

• it's difficult to have willing volunteers and not get them the right guidance
• and to make "eventually" now

c:

• thinking we should ask Reini for a list of concerns, needs, and use cases
• then try to make a design that addresses more of those

Jerry:

• especially as he's about to leave on a two-week business trip
• sounds like a PDD to me

Allison:

• sounds like several

Jerry:

• library loading, runtime components

c:

• does he have commit access?

Jerry:

• we can encourage him to apply

c:

• thinking about bindings to Prophet
• probably client-side

Jerry:

• we need sockets

Will:

• we have experimental sockets
• we need to make tickets or mark up PDDs for unimplemented things
• I did some of that with PDD 19 today
• stuff's deprecated and only exists in the PDD
• just to warn people who are going to write code that docs don't match reality

Jerry:

• that merits a CAGE task for reviewing the PDDs
• we could spread that out among many people

Will:

• I'll send an email

c:

• and post it on parrotblog

Will:

• no one reads those

c:

• they won't if we don't publish them

Jesse:

• I see them posted places like Reddit fairly often

Allison:

• probably will roll that into parrot.org eventually