All the Perl that's Practical to Extract and Report

Navigation

use Perl Log In

pmichaud (6013)

pmichaud (email not shown publicly)http://www.pmichaud.com/Patrick Michaud is the pumpking for the Rakudo Perl 6 compiler. He holds a Ph.D. in Computer Science and was formerly a Professor of Computer Science at Texas A&M University-Corpus Christi. He is currently a software developer and consultant focused on open source development and applications, including Perl, PmWiki, and Linux.

Rakudo Star 2010.08 released

[This announcement was made last week on rakudo.org -- I'm reposting to use.perl.org so it will show up in the various Perl aggregators. --Pm]

On behalf of the Rakudo and Perl 6 development teams, I'm happy to announce the August 2010 release of "Rakudo Star", a useful and usable distribution of Perl 6. The tarball for the August 2010 release is available from http://github.com/rakudo/star/downloads.

Rakudo Star is aimed at "early adopters" of Perl 6. We know that it still has some bugs, it is far slower than it ought to be, and there are some advanced pieces of the Perl 6 language specification that aren't implemented yet. But Rakudo Perl 6 in its current form is also proving to be viable (and fun) for developing applications and exploring a great new language. These "Star" releases are intended to make Perl 6 more widely available to programmers, grow the Perl 6 codebase, and gain additional end-user feedback about the Perl 6 language and Rakudo's implementation of it.

In the Perl 6 world, we make a distinction between the language ("Perl 6") and specific implementations of the language such as "Rakudo Perl". The August 2010 Star release includes release #32 of the Rakudo Perl 6 compiler [1], version 2.7.0 of the Parrot Virtual Machine [2], and various modules, documentation, and other resources collected from the Perl 6 community.

This release of Rakudo Star adds the following features over the previous Star release:* Nil is now undefined* Many regex modifiers are now recognized on the outside of regexes* Mathematic and range operations are now faster (they're still slow, but they're significantly faster than they were in the previous release)* Initial implementations of.pack and.unpack* MAIN can parse short arguments* Removed a significant memory leak for loops and other repeated blocks

This release (temporarily?) omits the Config::INI module that was included in the 2010.07 release, as it no longer builds with the shipped version of Rakudo. We hope to see Config::INI return soon.

There are some key features of Perl 6 that Rakudo Star does not yet handle appropriately, although they will appear in upcoming releases. Thus, we do not consider Rakudo Star to be a "Perl 6.0.0" or "1.0" release. Some of the not-quite-there features include:* nested package definitions* binary objects, native types, pack and unpack* typed arrays* macros* state variables* threads and concurrency* Unicode strings at levels other than codepoints* pre and post constraints, and some other phasers* interactive readline that understands Unicode* backslash escapes in regex character classes* non-blocking I/O* most of Synopsis 9* perl6doc or pod manipulation tools

In many places we've tried to make Rakudo smart enough to inform the programmer that a given feature isn't implemented, but there are many that we've missed. Bug reports about missing and broken features are welcomed.

See http://perl6.org/ for links to much more information about Perl 6, including documentation, example code, tutorials, reference materials, specification documents, and other supporting resources. An updated draft of a Perl 6 book is available as in the release tarball.

The development team thanks all of the contributors and sponsors for making Rakudo Star possible. If you would like to contribute, see http://rakudo.org/how-to-help, ask on the perl6-compiler@perl.org mailing list, or join us on IRC channel #perl6 on freenode.

Rakudo Star releases are created on a monthly cycle or as needed in response to important bug fixes or improvements. The next planned release of Rakudo Star will be on September 28, 2010.

Rakudo Star - an "early adopter" distribution of Perl 6

On behalf of the Rakudo and Perl 6 development teams, I'm happy to announce the July 2010 release of "Rakudo Star", a useful and usable distribution of Perl 6. The tarball for the July 2010 release is available from http://github.com/rakudo/star/downloads.

Rakudo Star is aimed at "early adopters" of Perl 6. We know that it still has some bugs, it is far slower than it ought to be, and there are some advanced pieces of the Perl 6 language specification that aren't implemented yet. But Rakudo Perl 6 in its current form is also proving to be viable (and fun) for developing applications and exploring a great new language. These "Star" releases are intended to make Perl 6 more widely available to programmers, grow the Perl 6 codebase, and gain additional end-user feedback about the Perl 6 language and Rakudo's implementation of it.

In the Perl 6 world, we make a distinction between the language ("Perl 6") and specific implementations of the language such as "Rakudo Perl". "Rakudo Star" is a distribution that includes release #31 of the Rakudo Perl 6 compiler [1], version 2.6.0 of the Parrot Virtual Machine [2], and various modules, documentation, and other resources collected from the Perl 6 community. We plan to make Rakudo Star releases on a monthly schedule, with occasional special releases in response to important bugfixes or changes.

Some of the many cool Perl 6 features that are available in this release of Rakudo Star:

Perl 6 grammars and regexes

formal parameter lists and signatures

metaoperators

gradual typing

a powerful object model, including roles and classes

lazy list evaluation

multiple dispatch

smart matching

junctions and autothreading

operator overloading (limited forms for now)

introspection

currying

a rich library of builtin operators, functions, and types

an interactive read-evaluation-print loop

Unicode at the codepoint level

resumable exceptions

There are some key features of Perl 6 that Rakudo Star does not yet handle appropriately, although they will appear in upcoming releases. Thus, we do not consider Rakudo Star to be a "Perl 6.0.0" or "1.0" release. Some of the not-quite-there features include:

nested package definitions

binary objects, native types, pack and unpack

typed arrays

macros

state variables

threads and concurrency

Unicode strings at levels other than codepoints

pre and post constraints, and some other phasers

interactive readline that understands Unicode

backslash escapes in regex <[...]> character classes

non-blocking I/O

most of Synopsis 9

perl6doc or pod manipulation tools

In many places we've tried to make Rakudo smart enough to inform the programmer that a given feature isn't implemented, but there are many that we've missed. Bug reports about missing and broken features are welcomed.

See http://perl6.org/ for links to much more information about Perl 6, including documentation, example code, tutorials, reference materials, specification documents, and other supporting resources.

Rakudo Star also bundles a number of modules; a partial list of the modules provided by this release include:

Blizkost - enables some Perl 5 modules to be used from within Rakudo Perl 6

MiniDBI - a simple database interface for Rakudo Perl 6

Zavolaj - call C library functions from Rakudo Perl 6

SVG and SVG::Plot - create scalable vector graphics

HTTP::Daemon - a simple HTTP server

XML::Writer - generate XML

YAML - dump Perl 6 objects as YAML

Term::ANSIColor - color screen output using ANSI escape sequences

Test::Mock - create mock objects and check what methods were called

Math::Model - describe and run mathematical models

Config::INI - parse and write configuration files

File::Find - find files in a given directory

LWP::Simple - fetch resources from the web

These are not considered "core Perl 6 modules", and as module development for Perl 6 continues to mature, future releases of Rakudo Star will likely come bundled with a different set of modules. Deprecation policies for bundled modules will be created over time, and other Perl 6 distributions may choose different sets of modules or policies. More information about Perl 6 modules can be found at http://modules.perl6.org.

Rakudo Star also contains a draft of a Perl 6 book -- see "docs/UsingPerl6-draft.pdf" in the release tarball.

The development team thanks all of the contributors and sponsors for making Rakudo Star possible. If you would like to contribute, see http://rakudo.org/how-to-help, ask on the perl6-compiler@perl.org mailing list, or join us on IRC #perl6 on freenode.

Rakudo Star releases are created on a monthly cycle or as needed in response to important bug fixes or improvements. The next planned release of Rakudo Star will be on August 24, 2010.

Rakudo Star (a "usable Perl 6") to be released by July 29

As many of you know, last summer we announced that we would be releasing
a "usable release of Rakudo Perl 6" to be called "Rakudo Star" in the
second quarter of 2010. We later refined our target release date to
be April 2010.

Until March of this year we were well on track to meet the April
2010 release date, but then I had an
family
medical emergency
that took me away from Perl 6 development. As a result of my situation,
the Rakudo and Perl 6 team met online in early March and decided that
an April release date would be unrealistic, and we instead
focused our efforts on trying to make a June release for Rakudo
Star, to keep with our original "second quarter 2010" goal.

Ultimately it ended up being twelve weeks before I was able to return
to active Perl 6 development (i.e., late May). During my absence the
others on the Rakudo and Perl 6 team made incredible progress on
Rakudo Perl 6; I think their progress shows that a truly capable (and
growing) team of developers has coalesced around Rakudo Perl. Thanks
to their efforts, as of late May the compiler had nearly everything
we identified as critical for Rakudo Star in the ROADMAP, with only a
few key features blocking on my personal participation. We therefore
felt we still had a good likelihood of meeting the June 2010 target,
and continued to work with that goal in mind.

As part of planning this week's Parrot and Rakudo releases,
we all met online to solidify our plans for the Rakudo Star release.
After much discussion, we decided that although we could likely make some
sort of Rakudo Star release in June, there was too much risk that
releasing in June would fall well short of our vision of what
we want Rakudo Star to be.

Therefore, we've decided to to let the release date slip one more
month and release Rakudo Star not later than July 29, 2010.
We are firmly committed to the July 29 date; whatever we have
available then, that's what we release. I know that another delay
will be frustrating to many (it is very frustrating to me),
and that some will undoubtedly cite this delay as yet more "evidence"
that there will never be a release of Perl 6. But given the
circumstances, I think we feel that we promote Perl 6 better by
moving the release date by a month than we would by releasing
something less than our vision.

For those who might claim that we should "release early", we
are still continuing to make regular monthly compiler releases.
The most recent release
(#30, "Kiev")
comes with a lot of improvements
over previous releases, and I truly expect the next release
(#31, "Atlanta") to continue the trend. As always, we continue to
invite people to try out the compiler releases and to visit the
Perl 6 website to see what
Perl 6 is already doing today.

Finally, on a personal note, my wife and I sincerely appreciate
the ongoing support, prayers, and understanding we have received from
the Perl community (and especially the Rakudo and Perl 6 teams) during
these difficult times. While my wife is still not "out of the woods" yet,
things are far better now than they were in the Spring, and we
continue to work towards and pray for her full recovery.

More details about the Rakudo Star release will be forthcoming
over the next couple of weeks.

An Anonymous Monk on perlmonks.org recently started a discussion on
"The current state of Perl 6" in which he (she? they?) offers his
views on various aspects of Perl 6 development. Personally, I
find nearly all of Anonymous Monk's comments in the thread to be
so devoid of logic and so far removed from reality/facts that it's
hard to imagine providing an adequate reply to them.

Beyond that, I have other priorities in my life right now.
Uppermost is that my family is going through a very difficult
period at present -- I need to take care of them first. Rakudo and
Perl 6 come in at a somewhat distant second to that. Responding to
garbage being spouted by anonymous person(s) probably ought
to not even be anywhere on my radar.

But at least one post in the thread bugs me sufficiently that
I'd like to respond, even if it's an inadequate response.
And I don't want my response buried in a thread somewhere, so it
gets its own post.

Oh I'll tell you how you do that [write a grammar engine capable
of supporting Perl 6]. It's very simple. You get people skilled
for this exact task ! Those skills are acquired in
universities(preferably good ones) where professors teach
courses like "Formal Languages and Automata" or "Compiler
theory". If you have a bunch of people who are open-source
contributors but don't have the knowledge or haven't studied
the right things... [t]hey don't know it in theory
and they're trying to put it in practice!
(emphasis in original)

Who does Anonymous Monk think is working on Perl 6? Personally, I
have a Ph.D. in Computer Science and taught university language
courses for nearly 14 years. Damian Conway certainly also
has a fairly strong university background (Monash University) and
understands the theory and practice of language and compiler design.
Jonathan Worthington has a degree from the University of Cambridge,
where he specialized in compiler construction and formal languages.
Larry Wall has a degree in Natural and Artificial Languages
with two years of graduate work in Linguistics at U.C. Berkeley.
Many of our other key contributors also have degrees and backgrounds
in language theory and practice. I'd be willing to compare the
academic credentials of this group against any other dynamic
language team Anonymous Monk might care to postulate.

If you can't get real compiler people
to use Perl6 and help with it, the average open-source rookie
won't be able to deal with this.

Somehow I have trouble classifying Larry, Damian, Allison, etc. as
being "not real compiler people". And they along with many other
Perl 6 contributors have a lot of experience in nurturing open
source developers and projects. If you feel Larry et al. are
indeed unqualified to be working in the field of programming
language design and implementation, you probably don't want to
be using Perl at all, much less Perl 6.

People like Anonymous Monk spew lots of opinions about how long
they think Perl 6 development should take, and then speculate on
the reasons why it has taken longer than they estimate it should
be taking. The speculations that crop up most often are things
like "the people involved are clueless about language
design/implementation" (see above), "the design process itself
is flawed", "they're building on an inadequate toolset like Parrot",
etc. For such people it's easy to toss out random thoughts about
"the problems with Perl 6" without bothering to look at
the obvious evidence to the contrary, as Anonymous Monk
does above. Indeed, I'm often amused when people suggest that
we should be doing things
that we're already doing.

Returning to the topic of developing a grammar engine, which
Anonymous Monk claims above as "very simple" and just needing
"people skilled for this exact task", it's interesting to contrast
his opinions with the actual development of the Perl 6 standard grammar
(STD.pm6).
I think STD.pm6 is also indicative of the challenges confronting all
Perl 6 implementors. Consider:

Larry is the primary implementor for the standard grammar.

The standard grammar doesn't need a frozen specification
to be implemented, because its implementation is considered
part of the spec.

The implementation is built entirely on top of Perl 5 -- there
are no "unproven virtual machines" involved or holding things
back.

I think one could consider this to be an almost perfect situation
for developing a new language implementation -- an experienced language
designer/implementor working without significant external
restrictions on top of an advanced programming platform like
Perl 5. Yet it's been three years since Larry started working
on this implementation of the standard grammar and parser, and
while STD.pm6 is very powerful, it's also certainly not "finished",
and has been through a number of significant refactors in its
development. This says to me that the amount of time and effort
involved is more likely due to the sheer audacity and ambition of what
Perl 6 seeks to accomplish.

(Some will undoubtedly look at the above and knee-jerk respond
with something like "If it's taken three years just to create
the parser, then finishing a compiler will take far longer still
and therefore Perl 6 will never see the light of day."
This argument ignores the fact that other pieces
are being implemented in parallel with Larry's work, that
software development is not a strictly sequential process, and
the obvious fact that there are already "working" Perl 6
implementations available.)

Anyone who thinks that Perl 6 is fundamentally based on
traditional compiler construction techniques taught in
universities frankly has no clue as to what a fundamental
paradigm shift Perl 6 represents to language design and
implementation. It's this fundamental change that ultimately
gives Perl 6 its power, but it's also why Perl 6 development
is not a trivial exercise that can be done by a few
dedicated undergraduates. As
TimToady on #perl6 says,
"We already have lots of those kinds of languages."

Personally, I'm impressed and honored to be associated with
the people who are working on Rakudo and Perl 6. I understand
that people are frustrated (and even feel burned by) the long
wait for something as cool as Perl 6; I share the frustration
and like to think that I'm doing something constructive about it.
But I also find the vast majority of suggestions, comments, and
conclusions coming from Perl 6's anonymous detractors to be
(1) things we've already done or are doing, (2) ungrounded in
reality, (3) in direct contradiction to reasonably observable
facts, or (4) attempts to discredit a product they have little
interest in ever using themselves. And far too many of the
comments, like the ones I've highlighted in this post, are
so easily refuted with just a little bit of fact digging and
common sense, it's often hard to believe anyone can be seriously
making them in the first place. Yet there they are.

Returning to my original theme, I think my response here is
inadequate because it leaves so many other of Anonymous Monk's
claims in the thread unrefuted. I could undoubtedly spend many
more hours analyzing and responding to the many fallacies and untruths
in the thread, but frankly I don't believe that's the best use of
my time. People such as Moritz Lenz, chromatic, Michael Schwern,
and others are
also writing extremely well-reasoned posts refuting the garbage, for
which I'm grateful, but it's far easier for Anonymous Monk and his like
to spin garbage than it is for a small number of us to clean up after
it. And it does need to be cleaned up, otherwise it festers and
results in even more public garbage that someone has to clean up.

I hope that this post will at least encourage more people in the
Perl community to critically examine the things they hear and
read regarding Perl 6, especially when coming from sources with
no apparent standing or reputation within the community. And
maybe a few more people will even assist in publicly refuting
the garbage ("many hands make light work"), so that the sloppy
thinking, analysis, and dialogue that people like Anonymous Monk
post doesn't spread to infect all of Perl.

Pm

P.S.: Some may reasonably conclude from what I've written
above that Perl 6 is somehow "aiming too high", that our goals
should be scaled back to make something ready "right now".
I have two responses to this: (1) we are making things
ready 'right now', just grab any of the available packages and
start working and reporting bugs, and (2) there are already
'scaled back' versions of Perl 6 appearing and being used,
such as NQP or even the components that execute the standard
grammar. Some of these other projects, like NQP, are being
used for "real" programs and applications today; they're not
simply theoretical exercises or research projects.

Others often claim that all this effort on Perl 6 would be
better spent on improving Perl 5. In my experience, those of us
working on Perl 6 have absolutely no qualms with seeing Perl 5
improve and continue to grow -- we welcome it. Indeed,
many of the features appearing in Perl 5 today come directly
from ideas and things first attempted in Perl 6, and we're
genuinely happy to see that. But just because Perl 6 developers
also like Perl 5 doesn't mean that doing Perl 5 core development is
interesting to us, or that (in my case at least) we'd even be
qualified to do Perl 5 core changes. We aren't commodity programmers
that are interested in simply being unplugged from one project and
into some other project that others think is more worthwhile.
Personally, I'd prefer to see people who are really into Perl 5
improvements continue to work to make them happen, and that
the surrounding ecosystem continue to evolve to enable that
to happen more easily for more people. Indeed, this is my wish
for all open-source projects, even the ones I don't find
interesting or otherwise disagree with.

Tuesday March 16, 2010

11:32 PM

Bad news, revisited

For those of you who are wondering where I've been lately with Perl 6 and Rakudo development, here's the story. As many of you know, my wife was diagnosed in early 2008 with ovarian cancer and remission [1,2]. Over the past few weeks various tests have indicated a probable cancer recurrence; and on March 2nd my wife was diagnosed with a small-bowel obstruction requiring immediate hospitalization. Since then nearly all of my time and energy has been essentially dedicated to caring for her and her needs. As of this writing (Mar 16) we're still in the hospital and hope she will be able to return home in the next couple of days.

The most obvious question relating to Rakudo development is "How does this affect scheduling of the Rakudo Star release in April?" Unfortunately, I don't have a good answer for this at the moment, as I don't yet have a firm date of when her recovery might be sufficiently far along that I can devote significant time again to Perl 6. I could be available again in just a few days, or it could take many weeks.

For now I've asked the core Rakudo and Perl 6 developers to work out what they think will be the best course of action for Rakudo Star given my current situation. As far as I'm concerned, all options are on the table (e.g., delaying the release a bit), but I think that whatever is decided needs to be a consensus among the entire development team. I suspect we will collectively announce an updated plan sometime in the next few days.

As I did in 2008, I'll tend to limit any further public posts about the current medical situation to major developments. But if anyone has any questions, I'll be glad to answer them.

As always, my wife and I send our thanks to everyone for your prayers, support, and understanding.

This is a milestone report for the "Rakudo Perl and PCT improvements"Hague Grant. Rakudo Perl [1] is a Perl 6 [2] implementation built on topof the Parrot Virtual Machine [3].

The overall purpose of this grant is to design and implement thecomponents needed to align Rakudo Perl's parsing parsing more closelywith the current Perl 6 specification and to increase the level ofPerl 6 code used to implement the compiler itself.

This report focuses on the completion of the deliverables neededfor milestones M1 and M2 in the grant description [4], and marksthe "halfway" checkpoint for the grant. Originally this checkpointwas expected to be reached in early 2009, but significant changesto Parrot, Rakudo, and Perl 6 in the months shortly after thisgrant was awarded delayed progress on some M1 and M2 items untillater in 2009. However, progress on many other items in thegrant were on time or accelerated, such that even though thisis officially the "halfway" checkpoint report, in reality almostall grant deliverables are completed, and the "final" reportfor this grant can be expected in December 2009.

A significant event during this grant period has been the planningand announcement of "Rakudo Star", a "usable and useful Perl 6 release"to occur in April 2010 [5]. Rakudo Star is not intended to be acomplete implementation of Perl 6, but rather to be an official"interim" release that provides sufficient features to be a viableplatform for consideration by application developers. Much of theoverall Rakudo project effort, including work to be performedunder this grant and other Hague grants, has been prioritizedand focused on ensuring the successful release of Rakudo Starin April 2010. Indeed, nearly half of the "critical must-have"features identified in the Rakudo Star ROADMAP [6] depend onthe deliverables from this grant; the current state ofprogress described by this milestone grant report is exactlyas planned and required by the Rakudo Star ROADMAP.

Items M1-3 and M2-4 concern handling of library precompilation andexternal interfaces; these were achieved in late 2008 and early 2009.Rakudo Perl allows library modules to be precompiled to Parrot.pirand/or.pbc modules for faster loading and execution; indeed, theTest.pm module is precompiled to substantially reduce the time neededfor Rakudo to run the Perl 6 official test suite ("spectests"). Manyapplications and libraries written for use with Rakudo similarlypre-compile the modules to achieve better load performance.

An interface for loading external libraries (including those writtenin other Parrot high level languages) was prototyped and added toParrot and Rakudo in early 2009; this interface is being continuallyrefined and improved in response to changes in the Perl 6 specification.

The remaining milestone items concern the implementation of protoregexesand longest token matching (LTM), and the integration of these featuresinto Rakudo. The initial expectation of the grant was that these featureswould be added to the existing Parrot Grammar Engine (PGE), andsubsequently used by Rakudo through PGE. However, as work progressedit became apparent that the changes needed to PGE would significantlyimpact backwards compatibility and run counter to Parrot's officialstability goals. Therefore, instead of modifying PGE I have builta new regex engine essentially from scratch and embedded the enginedirectly into a new version of NQP (the lightweight Perl6-likelanguage used to build the Rakudo compiler).

The new engine directly supports protoregexes (M1-1 and M2-1), alimited form of longest token matching (M2-2), and is much moreconsistent with the Perl 6 specification and STD.pm implementation [7]as they have evolved over the past couple of years. In addition,instead of compiling regular expressions directly to PIR (as PGE does),the new engine compiles regular expressions to the same abstractsyntax tree representation (PAST) used by other HLL compilers in Parrot.This allows better integration of regex support with higher levellanguages (e.g., HLL closures and variables embedded in regexes).It also may facilitate migrating the regex engine to backends otherthan Parrot at some point in the future. Another key feature isthat the new NQP and regex implementations are largely self-hosted-- that is, the source code is written in NQP itself, furtherenhancing maintenance and retargetability. The source code repositoryfor this new version of NQP and regular expression support iscurrently hosted on GitHub at [8].

Earlier this year Jonathan Worthington and I reviewed the tasksneeded for Rakudo to migrate to protoregexes and the standardgrammar (STD.pm), as well as achieve the other critical featuresneeded for Rakudo Star. Because the Perl 6 specification has evolvedsubstantially from when Rakudo's existing grammar and compilerwere first started, we decided that we would likely be moresuccessful rebuilding Rakudo from a "fresh grammar" (includingprotoregexes) rather than try to incrementally refactor theold grammar.

This rebuilding work has been proceeding in the "ng" ("new grammar")branch of the Rakudo repository, and thus far I am extremelypleased with our progress and results. The new grammar in thisbranch makes full use of protoregexes (M1-2 and M2-3), and isextremely consistent with the parsing model used by STD.pm .Furthermore, in the new branch we have already been able toimplement critical Perl 6 features (lazy lists, proper lexicalhandling, lexical subs) that were extremely difficult to achievein the previous implementation.

All of the items listed for milestones M1 and M2 of the grant havenow been realized. Over the next few weeks I expect to continuework on (re)building the Rakudo-ng branch; when it is passinga comparable percentage of the test suite as the existing Rakudoreleases we will officially redesignate Rakudo-ng as the mainlinedevelopment trunk. (The ng branch is effectively the mainlinefor development already -- we simply want newcomers to Rakudo to getthe "more complete implementation" in the old master branch insteadof the "rapidly catching up" version in the ng branch.)

In the process of completing this migration I also expect tocomplete the remaining deliverables (D1-D7) for this grant. Tobriefly review where we are with respect to each grant deliverable:

D1: (Implementation of protoregexes and longest token matching)
Essentially complete, with a few more improvements needed
for better longest token matching semantics inside of
regular expressions.

D2: (Alignment of Rakudo's parser and grammar with STD.pm.)
About 70% complete. Rakudo's grammar in the ng branch
is very close to STD.pm, with only minor additions and updates
(and redesignation of the branch as 'master') needed to
complete this task. It's worth noting that the goal for D2
is "alignment" of the Rakudo and STD.pm grammars as opposed
to "adoption"; indeed, in several areas STD.pm is changing
to reflect some of the ideas being pioneered by the Rakudo
grammar implementation.

D3: (Develop Perl 6 builtins for Rakudo, p6-based "Prelude".)
Complete. Perl 6 now uses the term "core setting" instead
of "Prelude" for its builtin classes and functions. Since
early 2009 Rakudo has increasingly relied on Perl 6-based
definitions of its builtin classes. In the Rakudo-ng branch,
most of the builtins not already written in Perl 6 are
placed in the "src/cheats/" directory in the expectation
that they will eventually be rewritten as Perl 6.

D4: (Develop and improve Rakudo's ability to use external libraries.)
Essentially complete, although further work is ongoing to
bring Rakudo and the compiler tools up-to-date with recent
changes to the Perl 6 specification in this area, and to
provide further documentation.

D5: (Continue developing official Perl 6 test suite.)
An ongoing task, but complete for the purposes of the grant.
At the time the grant proposal was written (September 2008),
the test suite contained approximately 7,000 tests and Rakudo
was passing 2,700 of these (38%). As of this writing, the
test suite contains over 38,000 tests and Rakudo master
passes over 32,000 (85%).

D6: (Create additional tests for regex engine and library components.)
About 50% complete. The NQP repository contains tests for the
new regex implementation and protoregexes, some additional tests
need to be written for the library interoperability components.

D7: (Publish weekly reports on Perl 6 implementation progress.)
Ongoing and on track. Although reports have not come out weekly,
there has been regular communication through our monthly release
process, updates to the Rakudo ROADMAP [6], the Rakudo twitter
feed [9], regular posts to use.perl and mailing lists, updates
to the perl6.org and rakudo.org websites, and other channels.
The goal of D7 has been to ensure increased visibility into
Perl 6 and Rakudo progress, this has been largely achieved
and will become even more apparent as we near the Rakudo Star
release in April 2010.

Of these, only #2 represents any significant or time-consumingeffort; the others are likely to be achieved in the normal course ofongoing Rakudo development. The remaining work on longest tokenmatching will be performed with the primary goal of meeting theneeds of the Rakudo Star release.

In summary, all of the items listed for milestones M1 and M2of the grant have now been realized, and many items in theremaining M3 and M4 milestones have also been completed. Thefew remaining deliverables for this grant are expected tobe achieved before the end of 2009.

The Rakudo-ng branch

As I wrote a few weeks ago,
the period since Rakudo's October release has been one of a lot of
refactoring and rebuilding to a new grammar and core implementation. So far everything
has gone almost exactly as I outlined at the end of that post, so this
is just a status update to let people know where we are.

The new version of NQP ("nqp-rx") has been substantially completed
and includes many new features that weren't available in the previous
regex engine. In particular, Perl 6 grammars can now support
protoregexes, and we also have the ability to execute code blocks,
declare variables, and add parameters to regexes. The new version
of NQP is also bootstrapped, which means that the code for all of
NQP (and its regex engine) is mostly written in NQP.

On October 30, Jonathan and I started a
new branch
(called "ng" for "new grammar" or "next generation")
for refactoring Rakudo to use the new NQP and regex engine
implementation. Nearly all of our work since then has been in
this new branch, and I think it is going exceedingly well.
Today we were able to get the sanity tests and core compiler
to where it can again begin compiling and running the test suite.

Outwardly this probably seems like very slow progress for 10 days
of effort, but running the test suite doesn't tell the whole story.
In the process of getting to this point, we also knocked off several
of the critical features needed for Rakudo Star that weren't
possible in the previous version of Rakudo, including lazy lists,
proper array element vivification,
dynamically generated metaoperators, correct handling of
containers and constant values, better dispatch semantics, and a
whole host of other features that make the Rakudo core much
stronger and easier to work with as we move forward from here.

Not only that, but we're already at the level where we are
implementing builtin classes and methods in Perl 6 (as opposed
to PIR). And we're also taking this as an opportunity
to rewrite some of the previous PIR-based components into Perl 6.

So, over the next couple of weeks we will continue to focus on bringing the new "ng" branch up to the same level as the current "master" branch. When we do that, the "ng" branch will likely
become the new master branch. I'm still a little reluctant to
give a time estimate for when we'll make this switch, but given
our current momentum (and the ease in which things are coming
together) I suspect it won't be far off -- a few weeks at most.
More importantly, unless we hit any major roadblocks, I expect
we will have nearly all of the critical components needed for
Rakudo Star (and certainly the most challenging ones) implemented
by the end of December. We'll then use the remaining months
until April to add other important but less-critical features,
continue improving performance, and work on packaging and
documentation issues.

Most of the progress reports (and
Twitter messages) I'll be
writing over the next few weeks will be about progress in nqp-rx
and in the "ng" branch as opposed to the Rakudo "master" branch.
Stay tuned!

Pm

Tuesday November 03, 2009

01:25 PM

A brief report on progress

It's been two weeks since I last did a blog post; and in that couple of weeks there's been an incredible flurry of activity and development. In fact, so productive that I barely have had time to write about it, although that will happen soon.

As a simple indication of the speed in which things are moving, I thought I would just post a copy of my report to today's #parrotsketch meeting. This represents things that have been achieved since October 27, and what we expect to be doing this upcoming week.

* Rakudo stuff:** Started the new implementation of Rakudo based on the nqp-rx
engine, that is going very well.** Have a new implementation of Lists, Parcels and Arrays,
all of which can now have lazy semantics.** Fixed constants and containers handling, Rakudo no longer
allows assignment to constant values.** Implemented "real" assignment metaoperator parsing (e.g., &infix:);
Rakudo-ng now builds assignment metaoperators only when needed.** Changed subroutines to be stored with the & sigil.** Changed Rakudo operators to be "&infix:<+>" instead of "infix:+".** Have many of the sanity tests running again; Test.pm compiles
but doesn't run completely yet

What I'm doing this week:* Continuing to work on Rakudo-ng, get us running spectests again
and compiling setting (now called "CORE")* More minor nqp-rx updates, error message improvements and better
syntax checking* Profiling the regex engine to find some speed improvements

What I'm blocking on:* Useful programming time

As you can see, things are happening quickly, and we're all pleased with the progress. Together with Jonathan's work on dispatch, we appear to have overcome the two major hurdles identified for Rakudo Star. From here on out it's basically just fleshing out the rest of the implementation and adding feature coverage according to the ROADMAP.

I'll write more details soon; I have to get back to a few other tasks right now. Hope you enjoy reading the above list as much as I enjoyed writing it.

Tuesday October 20, 2009

11:56 PM

Hague grant work: the new regex engine and NQP

It's been quite a while since I've written any articles about
Rakudo's progress, but the delay in articles has been because
I've been really focused on code development for a number of
things we're going to need quickly for
Rakudo Star.

At long last I've made the time to make substantial progress on my
Hague Grant,
which will enable us to bring Rakudo's grammar and parser much
more in line with the current
STD.pm grammar.
In fact, looking at the
Rakudo ROADMAP
one can see that a significant number of the critical tasks needed for
Rakudo Star are depending on the "PGE refactors" identified in the
grant.

This brings me to one of the major points of this post:
In the weeks that follow this month's release we expect that
Rakudo will be quite unstable as we undertake some much-needed
refactoring and redevelopment of some of Rakudo's core pieces.
The biggest change will be a complete replacement of Rakudo's
underlying grammar; the grammar we have today is still largely
based on the Perl 6 grammar as it existed in January 2008, but
STD.pm and the Perl 6 specification have evolved significantly
since then.

Jonathan and I believe that now's the time
to bite the bullet and make another big refactor to bring Rakudo
in line with the spec, even though it will likely involve a
rework of many features and perhaps a few significant
(but temporary) regressions. So, if you see some chaos and
upheaval in Rakudo development in the next few weeks, it's
a planned and necessary sort of mayhem.

Many of the needed grammar changes will be possible because
of the grant work on protoregexes and a new operator precedence
parser. Originally the plan was to build these features into
the Parrot Grammar Engine (PGE), but after thinking long and
hard about it I concluded that it would
be better to redesign and reimplement a new regex engine than
to try to fix PGE. For one, I think maintaining backwards
compatibility would be a significant challenge (and a drain
on my energy and resources). Another reason favoring a
rewrite is that we now have better language tools available
for Parrot, and a rewrite can take advantage of those tools.

Thus, instead of compiling directly to PIR, the new regex
engine compiles to Parrot's abstract syntax tree
representation (PAST). In addition, the source code
for the regex engine is written in NQP instead of PIR.

For those not familiar with NQP, it's a Perl 6-like language I
designed for Parrot in conjunction with the Parrot Compiler
Toolkit. NQP acts like a "mini Perl 6", it understands a
subset of Perl 6 language constructs and can generate Parrot
code that doesn't rely on additional runtime libraries.
Most of the HLL compiler authors for Parrot have been using
NQP to generate PAST, and it's proven to be much easier to
write and maintain than PIR.

Since the regex engine will now be written using NQP, it
also seemed fitting that NQP would receive the ability to
use Perl 6 regexes and grammars directly. Adding regexes
and grammars to the NQP language means that a compiler writer
can write nearly all of the components (parser, ast conversion,
runtime libraries) using NQP. This is in contrast to the existing
setup that requires multiple languages and APIs.

The new version of NQP is currently called "nqp-rx" ("NQP with regexes");
I may come up with another name for the bundle but I'm somewhat
attached to "NQP". This new version also has a new source code
repository (separate from Parrot) -- it's hosted on GitHub at
http://github.com/perl6/nqp-rx .

Since mid-September I've been working on nqp-rx, and I'm very
pleased with how it's all coming together.

For example, late last week I completed most of the work on
the new regex engine. This first version includes a very naive
implementation of protoregexes, which PGE lacked, and ultimately
should perform pattern matching and parsing more efficiently than PGE does.
It now compiles to PAST instead of directly to PIR, which means
it will fit more cleanly with the rest of Rakudo, especially
with being able to handle lexical variables and code blocks in
regexes.

More importantly, the regex compiler is self-hosted (or
"bootstrapped"). In other words, the regex engine is able to
parse and compile the specification that was used to build
itself. Stated another way, the regex engine is written
using Perl 6 grammars and regular expressions that it knows
how to compile.

Since completing the regex bootstrap I've been working on
creating the new version of
NQP based on the new regex engine. Over the weekend I created
some common rules for parsing number and quoted string tokens,
and yesterday and today I completed a new operator precedence
parser (all of these based on the STD.pm equivalents). Now
all of the pieces are in place to create a new NQP compiler,
which I plan to do over the next couple of days. And, like
the regex engine, I'm planning to make this new version of
NQP self-hosted as well.

So, when all of this is completed, NQP will continue to be a
"Perl 6 lite" language, but it will also support grammars,
regular expressions, protoregexes, longest token matching,
a very powerful operator precedence parser, attributes on
classes, and a fair bit more. It should
also be a bit faster than the previous NQP, and have a few
additional optimizations (such as inlining of blocks).

Thus, here's a quick rundown of the status and plan for
the next couple of weeks:

Scott Duff ("PerlJam") will be cutting the October Rakudo
release on Thursday, based on the Parrot 1.7 release.

Immediately following the Parrot release, new code for the
Parrot Calling Conventions is set to be merged to the
Parrot trunk. This is one of the major tasks (B) listed in
Rakudo's ROADMAP.

In the days following the Rakudo release, we'll be working
to synchronize the multidispatch and binding algorithms in
Rakudo with the new Parrot calling conventions.

Also in the next few days, we'll complete implementation of nqp-rx,
or at least bring it to the point that it can be used instead
of the previous compiler tools for building Rakudo.

When we're comfortable that the Parrot calling conventions
work has sufficiently stabilized, we'll start a new branch
for the major refactor of Rakudo's internals, switching to
the new compiler tools, and updating the grammar to be much
closer to STD.pm.

We don't know how long this last piece will take, but it
could easily occupy most of the month before the November
Rakudo release.

During the time that work is taking place in the branch, we
don't expect much progress or changes to be made in Rakudo's
master branch, so that people can continue to use and test
the "more functional" version.

If work bogs down in the branch, we'll regroup and come up
with an alternate plan. But I don't think this likely.

It looks to be an exciting couple of weeks! I'll be writing
more articles with details about the new regex engine, NQP
implementation, and Rakudo conversion to the new tools.
I hope and expect that by the November release we'll be
completely switched over to the new regex engine and
have knocked out a large number of the "critical" items
on the ROADMAP for Rakudo Star.