chromatic: March 2008 Archives

The Perl 6 design team met by phone on 26 March 2008. Larry, Allison, Patrick, Will, Jerry, Jesse, Nicholas, and chromatic attended.

Patrick:

• Perl 6 is going pretty well
• mainly catching up on mail
• finding out where people are and what they're doing
• I had a lot of mail
• pleased on the progress on Rakudo
• people are adding more features and looking at it
• still need to look at the new PCT tutorial
• my plan is to continue reviewing the new changes to Rakudo
• going to get back on track writing about what's going on

Larry:

• no spec work this week
• lots of conversations on p6l about various parsing issues
• relationships of unaries with indexes

Patrick:

• is it safe for me to ignore that thread?

Larry:

• for now
• whatever we come up with for the operator precedence parser in STD will be the standard way of doing it
• mostly fighting bugs in YAML and losing
• giving up on YAML for lexer storage
• it does not like to deal with Unicode

Jesse:

• is this a particular implementation of YAML?

Larry:

• all of them
• I'm using my own formatted file at the moment
• fighting bugs in tagged regular expressions
• it loves to coredump for no apparent reason, if you have too many alternatives
• in addition to the difficulties with null patterns and null character sets
• it always coredumps with those
• if I have 28 or so statement control alternatives, it'll run if I delete all but two or three of them
• apparently a forward or back pointer stored in a byte or something else badly
• may have to dive into the C code to fix that
• also finding a few Perl 5 bugs
• I really, really exercise lexical scoping within subroutines in my output
• nest maps very deeply and expects to keep straight lexical variables within those blocks
• something is leaking somewhere
• mostly working around those problems

Jerry:

• Summer of Code stuff now
• a few applications are coming in
• two look good for Parrot and Rakduo
• need to advertise for more
• also had a contact from someone who wants to port OpenGL to Parrot
• not Geoff Broadwell
• seems like a very serious approach
• pushing him to apply for funding from TPF
• also trying to keep on top of mail and catch up
• haven't had any coding time
• doing a lot more managing and answering questions
• to some extent, that's fine
• there are more people involved in the project

Allison:

• my brain is full of the strings PDD at the moment
• some substantial changes from Simon's original draft
• he had a good perspective
• I'm looking at overall architecture changes
• still supporting what we need to support
• just in a different way

c:

• started some Perl 6-related arguments online; it's been a while
• made a first patch that gives PIR profiling
• it's not a great approach
• it gives some visibility though, and I've found a few places for optimization
• found a ten-percent speedup in PGE in some cases
• Tcl spends most of its time parsing and re-parsing
• also going to go through the bug tracker again and see if we can clear out more stuff

Jesse:

• are you still thinking about applying Warnocked Perl 6/Rakudo patches?

c:

• unless Jerry or Patrick yell

Patrick:

• if you reply to them, I'll take a look at them
• I had architectural concerns about some of them
• don't want people cargo-cult things if we check them in
• but I'll respond to them if you find them and bring them up
• not all are in RT, some were just on the list

Richard:

• spent a couple of days at EclipseCon
• trying to get Perl 5 as a supported language within Eclipse
• working on a spec, and then we'll shop that around
• next week is a day trip to New York on a potential sponsorship call
• could be significant

Will:

• ripping out deprecated items
• hope to get everything we've deprecated out before the next release

Jerry:

• I'm in favor of that

Jesse:

• mostly having conversations about making progress this week
• lots of people are burned out
• we're not hitting milestones that make people feel like they've been productive
• I don't know that we have a good set of milestones in Perl 6
• nor that we could lay out a series of good, dated milestones for Rakudo

Patrick:

• I agree
• but you just keep working away and more things become available to more people
• one blocker is IO
• keeps coming up

Jerry:

• also exceptions

Patrick:

• Larry made a comment somewhere that the design is waiting on the implementations to figure out what they need
• we can go where we need to go
• but we can change later
• there's no point in waiting for now
• there's a draft design for IO
• having an implementation would help people do file IO in Rakudo
• I hear Perl is good about that

Allison:

• what do you need from Parrot?

Patrick:

• how's the IO PDD?

Allison:

• it's written but not implemented
• the implementation date is June, I think

Patrick:

• how about the basic stuff?
• open a file and stuff

Allison:

• that mostly stays the same
• you can start using that interface now
• I'll rip out the guts later when we start implementing the new system

Jesse:

• are you comfortable doing an implementation against what's there today?

Patrick:

• that's one of those areas I'd like to delegate

Jerry:

• I worked on the IO design
• I'm comfortable with Parrot's IO
• I'll read up on Perl 6's IO

Patrick:

• it'll make Rakudo more visible where people can use it and make something work
• reading from and writing to files will help

c:

• Haskell went a long time without that and it's pretty popular

Patrick:

• it comes up on the channel regularly
• we can write our own filters and stuff for test suites in Rakudo instead of Perl 5

Jerry:

• eating our own dog food

Patrick:

• do you have a feeling for the strings PDD delivery and implementation?

Allison:

• due for implementation June 1
• probably ready for rolling in for mid-June

Patrick:

• Rakudo is holding off on reading Perl 6 source as Unicode waiting for that

Allison:

• you can probably use some of Simon's optimization techniques in the PDD
• he defines a new string type
• you can use that before the full integration into Parrot
• gives you always a fixed width lookup
• as far as I can tell, that's what's expensive

Patrick:

• if I switched and translated everything over to UCS-2?
• I don't want to implement any C code personally
• what will exist in Parrot and when?

Jerry:

• let's lay on Simon to get something working soon

Allison:

• I can't guarantee that we'll have something before June 1
• but we can start implementing the new string type right away
• if we can get Simon to do a first draft, that'll help

Patrick:

• I just don't want to switch to a variable-width encoding, which'll make parsing really slow

Allison:

• if you transcode when something first comes in, you'll take a first hit but not subsequent hits

Patrick:

• the problem with transcoding to UCS-2 right now is that it requires ICU, and we don't have ICU on all platforms right now
• I could potentially add those operations...
• I did that for UTF-8

Larry:

• you might be able to use the Perl 5 program that spits out Unicode tables into Perl 5 friendly tables
• they turn into bitmaps in a way that you probably don't care about
• could use that to write something based on UCS-4 or UCS-2ish integers

Patrick:

• the UTF-8 code is directly based on those codepoints
• we work only with codepoints at that level

Allison:

• how much effort do you want to spend, knowing that the new string implementation is coming?

Patrick:

• the lack of Unicode support in Rakudo prevents the French angles

Jesse:

• how much of the Pugs test suite uses those?

Patrick:

• they don't show up much
• it's not a killer feature

Jesse:

• seems like you could go a long way without it being a problem

Nicholas:

• which codepoints do you need?

Patrick:

• in a case-insensitive search, we fold everything to a single case
• without ICU, when you hit a codepoint outside of Latin-1, Parrot throws an exception
• we check for downcasing first, which is slow

Jerry:

• or we could trap the exception

Patrick:

• but a downcase on the French quotes is a no-op
• I could catch it
• but it's a bit of painful overhead to add

Nicholas:

• with a UTF-16 implementation which matched downcase for Latin-1, would that work?
• or do people expect to use accented characters and have them work?

Patrick:

• short answer: yes
• right now, Parrot downcases ASCII, checks for ICU and downcases, and throws an exception for everything else
• one patch I have is smarter about the non-ASCII codepoint on the ICU part
• if it's Latin-1, then we can figure out how to do it
• that's pretty easy to downcase
• not that many additional codepoints
• if it's outside of that, we can throw the exception
• that range includes the French quotes

Allison:

• let's see if we can get Simon to do an initial implementation

Patrick:

• one of the milestones was documentation for PCT
• is the PCT tutorial close to that?

Allison:

• it needs to be in PDD form
• Will's talking to him about that

Patrick:

• I'm happy to work with him on that

Nicholas:

• have you been struggling alone with those bugs, Larry?
• have you had help from others?

Larry:

• when you have a bug in TRE from a DFA that's too large and you try to cut it down, and it goes away when you cut it down, I worry that I'll have to solve it on my own

Nicholas:

• AEvar wrapped it for Perl 5
• he may be familiar with it

Larry:

• it's down in the guts
• in the long run, we may abandon TRE and write our own DFA
• just a question if I can work around it right now
• TRE might need modification anyway
• it gives me the longest token, but it won't give me the second longest token if the first one fails
• not sure how to backtrack into that
• a parallel NFA might be more reasonable in that case

Nicholas:

• don't you need all the decreasing order of longest?

Larry:

• you make a list of all candidate token resolutions
• find the longest unique
• call that and hope it succeeds
• if not, and if you're not ratcheting, you need to try something shorter
• all the way down to nothing

Patrick:

• once you know the longest one, it's a lot easier to find the shortest one
• then you know the lookahead

Larry:

• that assumes you have a hash to look up the shorter keys in

Patrick:

• there's some value in knowing the longest one
• it'd be better to have an automoton in this case

Larry:

• I'd like to have this applicable beyond parsing and lexing
• any regular expression-like thing can automatically do DFA-style matching to the extent that it's reasonable
• and gracefully fall over to the other one
• there are various ways of hacking around it that would work for a lexer
• that's not the direction I want to go

Patrick:

• you're after bigger game

Larry:

• Perl originally integrated regex matching into the language
• we've ignored DFA-style matching for so long, we're late in integrating it
• but I think we can do it better than anyone else so far

Patrick:

• is that a new motto for Perl 6?

Larry:

• uh oh, another new motto!

Jerry:

• one thing in Rakudo stops us from writing Perl 6 methods and classes in Perl 6
• it's a bug or limitation in PCT, I think
• when you compile Perl 6 code to PIR to create bytecode you can call as a library, it creates subs with the same name as other subs
• the generator for the sub names starts with the same number in every file
• you'll get _BLOCK_10 twice
• will HLL and namespaces help that?
• does something else in PCT need modification for that?

Patrick:

• namespaces would do a lot for it
• I don't have a good answer for that
• the name generator needs a better universally-unique identifier for its names

Jerry:

• a UUID generator would go a very long way to solving this problem with Parrot in its current state
• anything we can steal?

Patrick:

• it'd be nice to have a good UUID generator in Parrot itself

Will:

• are they generated separately and then included in the same file?

Patrick:

• when I say "universally unique" I mean "I unique"

Allison:

• needs external tracking or something

Will:

• sounds like we're solving the wrong problem there

Patrick:

• if there's a way to make identifiers for subs static like in C -- file-scoped but don't leak out, we could use that
• I don't know if the :anon flag does that

Allison:

• that just makes sure that they don't get stored in the namespace

Patrick:

• we're typically talking about nested closures
• all in the same compilation unit
• as long as the PIR compiler can make all of those linkages such that there's no runtime symbol lookup, there's no problem

Allison:

• can we include the name of the sub the closure is in in the generated name?

Patrick:

• that may be a short-term solution

Jerry:

• if you put in the namespace and the name as part of the generated name, does that help?

Patrick:

• does the :anon flag gives us what we want?

Allison:

• it may not be enough

Jerry:

• we can test that easily enough
• I'd like to see this problem solved before conference season
• I want developers to be able to jump in and implement things in Perl 6 by then

Allison:

• seems like a more urgent problem than French quotes
• if it doesn't work, it's a bug in the current implementation of anonymous subs

Patrick:

• is there a ticket for this?

Jerry:

• if there isn't, I'll add one

The Perl 6 design team met by phone on 19 March 2008. Larry, Allison, Will, Mitchell, Jesse, and chromatic attended.

Larry:

• indisposed last week
• getting half a gallon of blood put into me in the hospital last week
• had various tests, all negative, so probably something not too terribly obvious
• going in for a lower endoscopy in a week or two
• will probably know more then
• probably something benign that's just leaking blood slowly
• I do feel better than I did a week ago
• it's amazing how much more quickly I can walk, when my blood doesn't think that I'm up at 14,000 feet altitude
• hopefully thinking more clearly too
• or at least less unclearly
• mostly getting caught up and doing some spec clarifications about the semantics of list assignment to arrays
• what you can or can't assume about the batching of asynchronous read operations
• did throw in one new feature this morning
• I wanted badly when I wrote a Perl 5 program last week
• there's no easy way to say "match as much of this string as you can, then pretend as if you succeeded, ignoring the rest of the string"
• it's very useful for matching abbrev. and such
• each subsequent character is optional, but only if the previous one matched
• nested optional characters

Jesse:

• we've had some horrible hacks to work around that

Larry:

• I put in syntactic sugar for that
• we'll see how it works in practice
• tried to figure out which rules can be rewritten and which can't, reliably
• thinking about the future of the bootstrap I've been working on with standard 5
• trying to figure out what the blockers are on bootstrapping
• next travel is to YAPC::Asia
• will stop in Seattle

Mitchell:

• struggling to get to a usable Perl 6 implementation
• tried to use the Rakudo parser and dump the parse tree, to go to a separate backend
• making pretty good progress
• the parser is a bit slow, though
• 40 seconds to parse 700 lines of code
• not quite tractable
• back to looking at a hand-transliteration of STD.pm into Ruby
• it's starting to gel
• may be a new attempt this week to use a slapped on backend to get to a usable Perl 6
• whenever standard 5 matures, it can be swapped in for the Ruby parser

Jesse:

• is there anything anyone else can do to help you on this?

Mitchell:

• once it can do Perl 6, there's a world of possibilities
• until it actually runs, there's not much for anyone else to do

Allison:

• working on the Unicode/character set PDD
• gradually changing its name to just "strings"
• Simon had a very good first draft

Jesse:

• does he know you're working on it?

Allison:

• I'll mail him
• gave a talk at Microsoft to their DLR/CLR groups about Parrot
• then met with their open source lab
• they'll do smoke testing for us on several different versions of Windows

c:

• we released Parrot 0.6.0
• most of the big changes are in the PMC PDD, which landed
• we've also had a big push to close new and old bugs in our system
• should give us a lot of energy in the near future
• right now I'm looking at a way to profile Parrot apps at the PIR level
• good optimizations available there only after we can measure them

Mitchell:

• did you look at KCachegrind?

c:

• I did, also the Devel::DProf format
• the big work so far seems to be instrumenting Parrot appropriately to detect spots where we need to measure things
• haven't yet decided which output format to use
• Valgrind is necessary if you use the Cachegrind format from KCachegrind

Will:

• TPF is now an official part of GSoC 2008
• haven't seen any Parrot or Perl 6 specific hits for the Summer of Code
• I'll send out something to the Parrot and compiler lists trying to recruit students

Mitchell:

• is there a way to send mail to Perl Mongers groups?

Will:

• I think Eric sent mail to the monger managers
• are there any Perl 6 or Parrot talks that aren't on the schedule yet for YAPC?

Jesse:

• maybe "getting started with a little language"

c:

• or "implementing a Perl 6 feature"

Will:

• I'll send mail to the lists

The Perl 6 design team met by phone on 12 March 2008. Allison, Jerry, Will, Mitchell, Nicholas, Jesse, and chromatic attended.

Allison:

• concurrency summit was very eye-opening
• the discusses circled around and came close to what I concluded for the Parrot concurrency model
• launched the PDD 17 implementation
• a lot of this implementation was cleaning up old, bad PMC implementations
• it shakes out old bugs
• gives us a chance to do some general cleanup
• working on OSCON now for the next few days

Jerry:

• haven't had much time to make commits this week
• just a couple of fixes to Tcl
• good to see that working so well on Windows
• working with Eric on Google's SoC
• he sent off the application earlier this week
• we're waiting for the good news
• Patrick's getting married on Friday
• next week, he and Paula will be traveling around the southwest, generally having a fun vacation

Jerry:

• there's a thread on the mailing list about setting up a Parrot workshop at YAPC::NA
• Patrick plans to submit a few talks
• I may too

c:

• I rewrote a lot of PMCs
• that helped us do the merge
• did some profiling, made some optimizations
• looking into an alignment bug with GCable elements
• Andy Dougherty and Jerry are seeing it on Solaris and Windows, respectively

Will:

• new Parrot monthly release next Tuesday, 0.6.0
• includes all of the PMC updates
• a lot of languages are in a better state now than they were before the merge
• I did some Tcl work recently

Nicholas:

• trying to work out what Perl 5 needs in terms of day-to-day bug flux
• might be useful to employ someone to handle that full time
• we have about a week's worth of bugs that comes in every week
• volunteers aren't doing that
• that'll be relevant to Perl 6 in the long term

Richard:

• TPF elections are underway, should be over in two or three days
• supporting Eric with the GSoC stuff
• lurking on the p5p discussion that Nick mentioned
• trying to figure out where the funding is coming from
• I'd like to get TPF to help out there
• discussing similar things with regard to Perl 6 in the past weeks
• those are positive
• they want to support people and get TPF to the place where it can disperse that support
• hoping that the elections help get TPF in better shape for that
• I'll be at Eclipse Con next week
• closed-door meetings with their brass
• they're supportive of TPF

Mitchell:

• my hacking's speculative
• looking at several pieces of Perl 6 implementations that have accumulated
• trying to find a way to combine them into a usable implementation to help an active P6 coding community re-gel
• sometimes trying to make it easier to access existing components
• sometimes trying to make them more accessible to reuse other parts
• we've had five or six attempts at full stack pieces and even more pieces, but none of them expose a usable parse tree (Pugs doesn't really have one)

Jesse:

• is it a small matter of programming, or are things blocking your way?

Mitchell:

• mostly my time now
• may be Parrot resources in the future

Jerry:

• we should coordinate talk submissions for YAPC

Jesse:

• is there a hackathon scheduled?

Jerry:

• just the workshop

Update: revised a note in Mitchell's report

The Perl 6 design team met by phone on 05 March 2008. Larry, Allison, Jerry, Will, and chromatic attended.

Larry:

• very little official work appeared to get done this week
• corresponded with Simon, Allison, and Patrick on a possible implementation of grapheme-oriented encoding
• looks good there
• no real spec work thanks to health issues, day job, and my daughter and son in law visiting
• working on the gimme5 translation of the standard grammar
• working on the correct way to extract matches, given a current match state
• how to present them to the user
• ... through the constraints of the Perl 5 type system which really dislikes having capture objects...
• went down one dead end
• digging my way out of that
• also working with putter, who's trying to pull together threads around the concept of a unified AST
• that's a valuable work
• trying to encourage him in that

Allison:

• launched the security PDD
• reviewed Simon's first draft of the character sets PDD
• includes an interesting new way of representing graphemes
• we've gone through various names
• UCS-4/NFG seems to be the most comprehensive name
• finalizing a few details now on the PMC branch
• we plan to merge that in on Saturday
• tracking down one segfault on Windows
• prevents PGE from compiling to bytecode

Jerry:

• focusing on Summer of Code applications
• I volunteered to manage Parrot and Rakudo projects
• submitted as part of TPF
• working with Eric Wilhelm, who really has his act together
• Will will be my backup
• good to have backups for all of our positions
• trying to offload some of my tasks to other willing hackers
• one of those is the YAML serialization
• hope to give that over to Will or Stephen Weeks this week
• spending the rest of my time debugging PDD 17 branch on Windows with Allison

Will:

• working on Tcl on the PDD 17 branch
• nothing particularly Perl 6-y

c:

• fixed the HLL segfaults
• fixed a segfault in the copy opcode
• this seems to have fixed most of Tcl and Lua
• did more work on the PDD 17 PMC update
• made a list of potential tasks for sponsored development

Jerry:

• does Moose buy you anything on Perl 5, Larry?

Larry:

• possibly
• can it easily produce an object which has all of hash, scalar, and array interfaces simultaneously?

Jerry:

• I don't know that offhand
• I know someone who does though

Larry:

• my problems are not so much the implementation as on the conceptual end
• getting my brain around what a match object is
• when you say make something or other, how much that replaces what's there and how much that augments the scalar part of what's there
• trying to think through that
• the main issue that I have is confusing the harvesting of subnodes with the stacking up of subnodes at the same level
• the latter should make lists of matches rather than embedding them
• just a matter of getting a translation to Perl that will do that once I've thought it through
• maybe Moose will help me there
• it's a good idea
• it might run slower
• my purpose in translating to Perl 5 is not to get a perfect translation but just to get one that will run the standard grammar
• doesn't have to be pretty
• only correct enough in the areas that matter to get it to run
• if correctness were a bigger goal, I'd certainly use Moose