What is a "Stable" Core Anyway? The Dual-Lived Problem.

A recent discussion on the p5p mailing list seemed like an opportunity for me to suggest what could help produce regular Perl 5 releases. (Nicholas Clark nearly dared me. He's a bad man.)

As you might expect, the resulting discussion went off in several directions. A few people see little value in regular releases (which means that I should write about that soon). Perhaps the most important point is a question that a couple of pumpkings alluded to. David Golden, ever perceptive, picked up on the real question.

Double Lives Take Half as Long

(I addressed this issue from the other side in February in Hanging the Core out to DRY. This is a followup from the "Why does so much time elapse between stable Perl 5 releases?" department.)

One of the biggest time sinks to releasing a new stable version of Perl is
chasing down dual-lived core modules. A dual-lived core module is a module in
Perl's core library that someone other than p5p maintains outside of the Perl 5
core repository. At various times, someone notices that there's a new release
of a core module (usually the maintainer, but...) and suggests that a pumpking
merge the new version into the core.

Why are there dual-lived modules?

The most important reason is so that users can upgrade to newer versions of core modules without upgrading the core. Why would they want to do that? Perhaps it's difficult to upgrade core Perl and they don't have permission. Most likely, it's because Perl 5 releases are infrequent. Where a bugfix in a core module may require only a little bit of testing and represent only a little bit of change, releasing a new version of Perl requires a lot of testing and coordination.

Another reason for dual-lived modules is to spread the maintenance burden. While only a few people had commit access to the Perl 5 repository in the past, every maintainer of a dual-lived module can have his or her own repository just for that module. This is less of an issue now that Perl 5 uses git, but old design decisions tend to persist.

A final reason for dual-lived modules is that the modules themselves may be
usable on several versions of Perl, even if they're not binary compatible.
(The modules may be pure Perl, or their XS may be source compatible with
multiple Perl major versions.) If code works correctly and unmodified on Perl
5.6.2 and Perl 5.8.6 - 5.8.9 and Perl 5.10, is there a reason to limit users to
waiting for Perl 5.10.1 or 5.12 to use the new version?

Of course, coordinating separate releases from all of these separate repositories and separate authors on a well-known and well-understood time frame and testing them to make sure they work with the core language at that point as well as with each other -- and remember there are dozens (if not more) dual-lived modules spread out among dozens of authors -- is difficult.

As I'm sure you realize by now, this is a tangle.

Cutting the Knot

These separate goals of dual-lived modules are incompatible.

The point of having a standard library full of modules is so that Perl is useful to users by itself, with nothing else installed. This theory suggests that users shouldn't have to install CPAN modules to do useful things.

The point of releasing modules on the CPAN is so that users can upgrade them independently of the core. The CPAN is an integral part of the modern Perl programming experience. (When users complain that they have to install "half of CPAN" to get a modern Perl application to run, part of that complaint is that Perl 5 the language is too minimalistic and flexible and not opinionated in several places to write modern Perl applications on its own. Another part of that complaint is that we don't have CPAN quite right yet.)

I can think of three solutions. They can overlap, but they're also
independent.

First, improve the core's automated testing. This helps everyone; it can identify changes in the core code that affect the stability and behavior of the standard library. It can also identify changes in standard library modules which do not work on important platforms.

The sooner you can discover a failure, the easier it is to identify the offending change.

Second, do not automatically merge in all upstream changes to the core. A stable and well-tested version of a dual-lived module is better than the most recent version, at least if you haven't verified it on all interesting platforms. If users want a newer version, they can upgrade themselves. (That's one of the reasons dual-lived modules exist!)

I realize this seems to conflict with my belief that users should upgrade frequently. I still believe they should. I still believe strongly in frequent stable releases -- but users do not have to upgrade. They should have the option to upgrade frequently (and they should know the support and deprecation implications of not upgrading), but they have the choice.

Third, remove dual-lived modules from the core. Maintain and distribute only those modules necessary to install other modules.

This will expose several other problems, however: encouraging Perl distributions to produce stable bundles which represent a usable core, improving configuration and installation support, and overcoming the social inertia toward an unchanging Perl core.

Tags:

2 Comments

The third option, removing dual-lived modules from the core, does seem like the most sensible. It comes with some interesting issues, though. Do we really want Test::Harness removed? Frankly, I don't think we've ever really given any thought to the impact we have on the core by our frequent releases, but there's an interesting tension here.

The core needs just those features that it needs from Test::Harness.
The Perl community needs more than the core needs from Test::Harness.

We push ahead with new releases of Test::Harness because of TAP archives, parallel testing, structured diagnostics, multi-language support, the upcoming nested TAP and so on. The community needs those new features but the core, while it could (and does) benefit from these innovations, could probably run quite happily on the old Test::Harness 2.64. We could try to rewrite it to provide Test::Harness::Extended (or some such) to separate out these needs, but it would be a daunting task and I'm sure that not being able to take advantage of these new features would be disappointing to some Perl 5 Porters. With testing being so crucial, do we really want to deny the core the ability to flex its testing muscles? How would you approach the problem?