rakudo spec test coverage chart

one type of question about rakudo i often receive at perl meetings and on mailing lists and irc channels goes something like, "are we there yet?" sigh. as a rakudo developer, it's about the journey as much as it is about the goal. however, we still need to measure our progress to the goal, in order to manage our priorities. the vague (and sometimes pointed) questions like that above help keep me focused on the rakudo team's first big milestone, which is to produce a release package with enough functionality so that rakudo "feels like" perl 6. i expect some subsystems can be loosely classified as "alpha" status, and others "beta". but how does the development team know when we're ready for release?

qualititive measures are all we have. the perl 6 spec is large, and incomplete. the spec test suite doesn't fully cover the synopses that have been written. the tests that do cover the synopses aren't organized well enough that we can quantitatively determine spec coverage by the test suite. greater community involvement can overcome these obstacles, but that's a different topic which i hope to address soon.

rakudo's tools/test_summary.pl program runs the spec test suite, and reports a summary of results by synopsis. it lists the number of tests in the suite, the number of regression tests rakudo expects to pass, passing, failing, todo, and skipped tests. those results help us get an idea of what bits of perl 6 design are working well. that info wasn't clear enough for me, so i've made two charts out of today's data.

the first embedded chart is below, and if you can't see it, you can click your way to the percentage chart and look at it there.

the chart lists percentages based on the number of spec tests for a synopsis. since we don't know how well covered each synopsis is by the test suite, this chart only goes so far. for example, in synopsis 11, we're passing 56+% of the spec tests. however, that's only 30/53 tests. certainly, we need more than 53 tests to cover the spec for modules.

so, here's the second chart, which lists a count of tests per-synopsis. again, if you can't view it embedded here, click through to the count chart and view it there.

finally, if you need help putting names to synopsis numbers, you can view a list of the synopses at http://spec.pugscode.org.