Implementing Perl 6... and other related technologies.

2005.09.30

Today's workshop has been a blast. David Roundy's darcs talk covered
many challenges and advantages in using Haskell to develop a real-world
application; he even mentioned SVK once, in a positive light. Among
other cool tricks, he employed interleave IO to separate program logic
with I/O parts. I have used that trick in OpenAFP, but I'm beginning to
see that has wider applications as a Pugs/Perl6 idiom -- AOP with Lazy Lists,
if you will.

The Cabal/Hackage system (the equivalent of Module::Build/CPAN) generated much interest. Discussions centered around how to incite spotaneous mutual goodwill using social software frameworks, which is imho the key part of the CPAN cultural mechanism.

I'm glad to hear that the huge GHC distribution will be broken apart into various packages, to promote the sense of community library ownership. In the future, the GHC core will be minimal, just enough to run Cabal to install packages -- Perl5 porters have bounced this idea for quite some time now, and it's nice to see it happen here too. In addition, the Haskell people are also reducing the committer-bit deadlock, by adopting decentralized darcs repositories as the norm, so a change of maintainership is simply a URL redirection apart.

Another focal point is hIDE, a Haskell development environment coded in Haskell, using the GHC bindings and build structures developed in the Visual Haskell effort. The choice of Yi as the editor component is particularly interesting -- it can dynamically load and rebind plugins, including the editor core itself, preserving the editor state (undo history/workspaces) without requiring a restart like Ecplise does. I suggested to dons that we should integrate the bootloader with GHC/Cabal, so all Haskell applications compiled this way can be code-swapped on the fly, even by eval'ing code from the user. I like dynamic languages with static typing...

I helped the library effort by polishing dons's Data.FastPackedString module, so it can subsume the old Data.PackedString implementation. The new string module
is very fast and supports FFI, gzip and mmap natively. It is already used in the new PIL runtime; once it's merged into GHC core, it should give string manipulation-heavy programs a nice speed boost.

Hacking of GHC core continued. Pairing with wolfgang, the new ICFP 2005 contest champion, is very exciting and fruitful. We reasoned out how to support record updators for existentially-quantified GADT types, which is starting to dangerously resemble ML-style modules.

Also, from discussion with oleg and ccshan, I learned that GHC is actually well-equipped to natively support delimited continuations in the runtime. If implemented, Pugs's Haskell runtime can do away with the space-intensive ContT monad transformer thunks, and get a massive speed boost without changing a line of code. simonmar showed me the GHC runtime scheduler, and indeed there seem to be native support in the form of thunk-capturing asynchronous exceptions (which will incidentally give native support for the Perl6 $!.resume). Using a similar mechanism, we can also get a stack trace -- easily the #1 requested debug feature -- from fatal errors, at least when compiled for profiling.

Whew. That's a lot of hacking. Tomorrow I'll go back coding in MM2 and full Cabal support, and investigate full continuations on top of Perl5 VM. Stay tuned!

2005.09.29

Continuing yesterday's work, today I got seriously addicted in GHC core hackery, skipping the entire Curry workshop. It was a real pleasure pairing with spj on this. He reminded me of lwall -- both have an obsession on providing useful feedback to programmer, strong intuition on how language features intersect, and above all a deep enthusiasm to their respective languages.

Thanks to spj's continuous guidance, I hacked in support for existential record types on both vanilla and GADT types, which allows us to write:

The tag function is well-typed as Counter public -> public; the other three fields hides the real implementation from the Counter data type, so one can have a counter using Int, another using String, and have both be of the same type.

This representation is quite close to how encapsulation works in OO langauges, and the feedback from #haskell has been very positive. Additionally, the soundness of the GADT/Labels treatment passed scrunity by Oleg today, which made me feel much safer.

Of course, any attempt to use the three private fields as normal functions raises a compile-time error that advises the use of pattern-matching syntax instead. Also, in the GADT form, the forall this may be omitted to improve readability.

While I was madly hacking away, brendan passed by, and graciously accepted a Pugs T-Shirt from me. We talked about many things -- I'm very excited to hear that they are considering to adopt Rhino's continuation engine for SpiderMonkey (which will make Perl6-on-JavaScript insanely faster), and that the XUL scripting is proceeding well. They would like to use user's system python/perl/ruby/etc if possible, but that complicates the versioning problem.

Inevitably, the topic of shared runtime came up, and it seems that Mono's RAND patent position is still not yet very satisfatory. Parrot has matured a bit since two years ago when brendan checked it out, so I filled him on some recent development. With 0.3.0 out in a couple days, we can finally start targetting Parrot again -- hopefully the long wait will be proven worthwhile.

Oh, do check out geoffb's writeup of his musings on #perl6; it's a nice survey of some of the good ideas from other language that are finding their way into Perl6.

After yesterday's discovery of ILX codegen, spj pointed out that there is a more robust Java codegen on the same directory. Targetting JVM suddenly looks more attractive too...

All in all, I think my time on hacking GHC core is well-spent; I learned a lot of Haskell idioms that should make Pugs code cleaner (such as the case x of DataCon {} ->... syntax that works even on non-record types), and about complexity management of a large real-world language implementation -- i.e., what makes it to evolve extremely fast, despite a relatively small core team and contributor base, and an utterly broken issue-tracking system. RT-powered bugs.haskell.org seems like a good idea...

Tomorrow is the Haskell workshop, the final day and highlight of this conference. See you!

2005.09.28

The main ICFP conference session concluded today, so I'm finally getting a
tiny slice of time to write down what I've been doing.

I enjoyed handing out Perl6/Timeline T-shirts to various lambdafolks,
including bpierce (author of TaPL), spj (author of GHC), as well as several cool
hackers from #haskell. At any time in the conference hall, there was always
someone with a Pugs shirt in sight, leading to many interesting discussions and
opportunities to join forces.

I spent much time hacking with the GHC core team (spj and simonmar, also known
as "Simon and Simon"), to get my current #1 feature request -- GADT with record
types -- into GHC. Among Haskell people, GADTs are all the rage this year; see
luqui's
notes for a short introduction from Perl6's perspective.

During the YAPC::NA 2005 Toronto hackathon, I presented the then-brand-new
PIL1 data structure to @Larry, using GHC's then-brand-new GADT syntax:

luqui looked at it and asked quite innocently: "Why are the arguments
not labelled, such as 'lhs' for 'PAssign'?" -- To which lwall said,
half-jokingly: "Because autrijus is a lazy bastard."

However, the real reason was because GHC's extended Haskell syntax only allowed
positional argument types for GADT. To name them with the record
syntax, we'd need to go back to the vanilla data types, and forgo the
ability to encode type information using the constructor's return type (as
demonstrated in PIL Expression and PIL LValue above).

A month later, I diligently re-encoded the structure using vanilla
record types, so we can export them into hash-based Perl (and JSON) data
structures, instead of array-based ones. This significantly improved
the readability of iblech's PIL2JS code, but the Haskell data declaration
became 3 times longer, because each GADT variant needs to be re-encoded
into vanilla data types.

Ever since then, I was hoping that GHC may one day remedy this by allowing
GADTs and record syntax to play together. When I met spj in ICFP, he was
gladly surprised that I'm eager to hack GHC to make it happen. After bouncing
several design ideas, we settled on this syntax:

The neat thing is that PBind and PAssign can share
the lhs and rhs labels, as long as their return
value PIL LValue stays the same.

During the reception, spj was kind enough to guide me through the GHC core to
unify the two data type definitions together back. The next day, simonmar
walked me through the Happy parser
to get the new syntax recognized, and today spj showed me how the GHC typechecker
works. All there's left is installing the labels as record selectors, and the
upcoming GHC 6.6 will have a nice feature to fit Pugs's needs. Hacking GHC is
evidently not as difficult as I feared!:-)

Talking about GADTs, I was also involved with the crazy project of encoding the
Darcs patch algebra into the GADT type system,
letting the compiler catch entire classes of common logic errors in darcs,
bringing us dagerously close to the ideal of proof-carrying code.

Two days ago, when droundy told me this idea, I was very excited, started
prototyping it, then promptly ran into errors on existential types. Heffalump
took my laptop and shuffled it with Igloo, arriving at something that seemed to
compile, which evolved into an embedded DSL for patches. After today's
conference sessions, a dozen darcs hackers gathered together, learned about the
recent developments of the patch theory, then worked on the GADT encoding.
The hacking session was quite successful -- we didn't run into any showstoppers,
much to everybody's surprise -- and I visiolized the whiteboard into this
diagram.

On the Pugs backend front, I learned from simonmar the existence of the ILX
emitter library, hiding in a forgotten corner of GHC source tree. This enables
us to emit pre-CLR-2.0 MSIL code that could run on Mono, under a 3-clause BSD
license. Targetting CLR suddenly looks more attractive...

I also learned how to generate native machine code straight from C--, using
the GHC API distributed as part of Lemmih's ghc-api Cabal package.

Speaking of Cabal, I learned from SyntaxNinja about the design of the Cabal/Hackage
system, which is the Module::Build/CPAN equivalent for Haskell. We shared our
experiences, and he helped me setting up the basic Cabal build rules for Pugs.
With Cabal support, Pugs's Makefile.PL would only need to care about the Perl
parts in the source tree, and we can finally get a libPugs to link with pugscc
and other programs -- Inline::GHC is definitely on the radar, and the idea of
using Perl6 to script the Haskell IDE (hIDE) is also gaining some traction.

Compiling with continuations, another pet interest of mine, also got a boost
from ICFP. The Generalized
Stack Inspection talk outlined how to use vanilla try/catch
mechanism to implement full continuations, which is much more efficient than
the regular trampolining approach as used in Pugs's JavaScript backend, since
only people who actually use full continuations need to pay the cost for
catch. Their technique operates on the ANF form, which seems to
be a good low-level intermediate form for Pugs.

Interestingly, the paper went as far as saying that Parrot might have made a
suboptimal choice in paying the full cost of going full-CPS, since we could
emulate full continuations using exception handling efficiently. They gave
concrete implementations in CLR and JVM -- we'll see if this result can carry
over to Perl5 and JavaScript runtimes.

The invited talk of the first day tackled impredicative type inferencing, a
long-standing problem in inferring types for a rich system such as Perl6.
I had much trouble grokking the Wobbly Type idea of spj's, so I
was much delighted to see that fpottier gave a very intuitive treatment of
splitting the user-annotation-driven phase and the complete inferencing
phase apart.

Oh, the ICFP contest. Haskell won again, which came at no surprise, but
the Dylan Hacker team made another strong showing. From the brief
investigation, I think Dylan is even closer to the "Perl 6 without
advertisement" mark than Ruby, largely due to its incremental soft
typing system, where one can start rapid prototyping with full dynamism,
then generate more efficient and robust C code by adding type annotations.

Tomorrow is the Curry workshop, then the Haskell workshop on the day after.
After that I'll get two weeks of Pugs hacking time. I can barely wait to
apply those new techniques, tools and thoughts into Pugs!

2005.09.25

...was very well received. The other talks in CUFP were very interesting too.

Talked with David Roundy about the svk-darcs isomorphism, and learned how to use darcs tags to simulate svk merge points and improve on two-way mirroring.

Daan's Morrow language is quite inspiring, especially due to its use of UUAG. I learned attributive grammars from the TMR article, and was happily surprised that @Larry seemed to be in favor of adopting AG as part of Perl 6's self-host compilation.

I'm totally sold to the Takahashi method for presentation. It takes 3x time to prepare but is worth it. The slides are available online as
odp,
pdf,
swf and ppt. Enjoy!

2005.09.23

So I'm in Zurich airport and (finally) working toward the slides for my talk, cheduled to go on air in 24 hours.

I'm adopting the Takahashi method, which basically means doing six lightning talks in a row, and somehow string them all back together in the end. It's fun and very challenging...

The State of the Onion 9 talk transcript is quite entertaining. I hope to run into other fun amplifiers in ICFP, so the two weeks after that can be optimised to get Pugs 6.2.10 (or even 6.28.0) out in a useful shape in time for Euro OSCON.

Perl6-MetaModel News

Role work is now (mostly) complete. We still need more tests, and there is
still more Class and Role integration planned but not yet implemented.

To exercise the meta-model I have ported chromatic's Perl 6 version of
Test::Builder (complete with tests) to the meta-model. The translation
was fairly seamless, and even used the meta-model based package system
instead of Perl 5 packages.

You can see it here
and you will find the Perl6 code in the POD documentation for reference.

Blondie wants You (pointing finger) to give it more backends

Blondie is a side project whose purpose is to prototype some approaches towards
compiling the dynamic/static conglomerate that is Perl 6 into something that
runs efficiently and portably, in as little effort as possible.

Blondie needs as many backends as possible as early as possible to stress it's
design.

Blondie is already interpreted in perl, and partially compiled to C (supported
by a primitive type inferrencer), but we want more!

For two reasons:
Running Perl 6 on foreign runtimes designed for high level languages
Learning what VMs look like when targeted uniformly

GCC's GENERIC

Although we're already harnessing GCC with the C backend, making a GCC
frontend for Blondie has two interesting aspects: it's higher level than a
VM, so this illustrates AST conversion, and it requires completely manual
memory management

Other interpreted backends, e.g. JavaScript

Interpreting approaches are by far the quickest way to get Blondie to run on
strange platforms. This will stress the way the shared prelude is handled.

Mock backends

Harsh environments for a dynamic language, like no IO, hard real time (all
builtins which cannot be realtime are masked from the prelude), etc.

Please see the todo file for more tasks under perl5/Blondie/TODO in the pugs
repository.

2005.09.21

Wow, I'm still alive after this tumulous week, which featured not one but three met-at-the-final-minute deadlines, a rescheduled and forced-upgraded plane ticket, my passport stuck in limbo for three days due to DHL system errors, and of course even more hardware failures.

At long last I'll be able to catch some sleep now, wake up and board the plane, step off it and sleep for a few hours, then immediately step up to speak in CUFP, with nary a page of slides in sight (or in mind). Wish me luck, as I'd need it very much...

Perl6-MetaModel News

Some re-arranging has been done,::Object now shares pneuma.pl with::Package and::Module, while::Role has been moved to the new
psyche.pl file. This means that genesis.pl is concerned simply with
bootstrapping things, and nothing more.

The Role work is progressing slowly becuase of $work. However, we have
added pictures (p6_role_model.jpg) and there is some general thoughts/comments in psyche.pl
as to how the bootstrapping will actually work.

We recently added a test which implements the inside-out style of
objects within the meta-model. This basically shows that the meta
model is capable of itself hosting other models.

Basic support (read: very experimental) for parameterized classes
has been added recently. This is implemented entirely on the "macro"
level, and not actually part of the meta-model itself. Here is an
example of how it is used:

2005.09.15

This means that %hash.kv[1]++ or @array.kv[1]++ modifies %hash or @array respectively. Additionally, the tests of.kv and.pairs were appropriately improved.

Perl6-MetaModel

Some basic work on Roles was done, and several new tests added for this. Mostly
just getting the basic Role object working as expected. Next step is to do Role
compositing and Role consumption by Classes.

cperl6-mode (Perl code editing commands for (X)Emacs)

Many new keywords/builtins, much better indentation,
better working of parameterized rx// modifiers, highlighting of sub
prototypes, non-parenthesized conditions for most statements, more
Perl6'ish sigil interpretation, dedicated "Pugs" style for pugs::hack
code conventions.

Blondie is a dumb intermediate language and a compiler collection for
the purpose of demonstrating an approach to compilation.
It has no syntax, only an abstract syntax tree, and some very simple
operations.

We decided today, after some discussion on #perl6 that class attributes need
not be handled by the metamodel, and could be just treated as package level
our variables. This of course simplifies the meta-model, which is always a
good thing.

The latest version of the object model diagram
was posted to perl6-langauge for review. This represents the current state of
the meta-model prototype, our next step is to add Roles.

Debian package updates

pugs 6.2.9 was recently uploaded to Debian unstable and will go into the
archive with the next dinstall run on 14.Sept 05.

The Debian package now Build-Depends on libghc6-plugins-dev. So now we have a
pugs with perl5 and haskell eval support. Now the only thing missing for a full
featured pugs in Debian is parrot, which is being worked on currently.
General Pugs updates

Pugs can now be called with a specific runtime engine. The Perl 5 backend is
invoked with pugs -B Perl5, and the JavaScript backend can be invoked with
pugs -B JS.

The $?PUGS_BACKEND variable tells which is the current backend.

The wrapper script around pugs_bin, pugs, was modified so pugs -BJS
respectively pugs -BPerl5 will launch the interactive shells of
PIL2JS/PIL-Run. This also means that make test-js works now.

2005.09.10

Mail is back to service, but only barely. Thanks to the helpful interface of a Sil RAID1 controller, I managed to low-level format all of autrijus.org's two disks, as well as the extra backup, which was selected as the target -- somehow the controller cheerfully formatted everything else too.

Combine this with the recent laptop HD failure, it means all I have left are publicly aggregated data and whatever archive.org has in it.

Among the definitely-lost category are mail archives, several active websites, and lots of unpublished works -- not the end of the world, but far from pleasant, either.

2005.09.09

The power supply of my home server (autrijus.org) burned up unexpectedly, possibly taking the motherboard with it. For the next 18 hours or so, mails should be able to reroute to autrijus at gmail.com automatically.

Also today I realised that Estonia does not take Schengen visa after all, which leaves me with ten days to get two visas done -- which normally would take more than one month. I'm trying to speed up the process so I can get a remote chance of presenting Pugs at CUFP.

So today's all spent on errands... Sorry for the lack of update again.:-/

Oh, btw, Edd Dumbill's interview with me was posted to perl.com today. Edd does know how to ask intelligent questions, but sadly I only had tuits to answer very briefly.