Airlines

I just got an email from one of the Airlines I fly on
notifying me of a terrific opportunity: they will now have
more seats in coach. In the email they seemed to be pretty
psyched about it. I mean, really excited about it they even
mention that they will continue to provide snacks.

The new HTTP application pipeline in Mono is in my opinion
a beautiful work of art. Let me explain.

The HTTP application pipeline processes an incoming HTTP
request. When the request comes in it has to go through a
number of steps before the request is actually handed over to
the developer code: authentication, authorization, cache
lookup, session state acquisition. There are a similar set of
steps processed after your code is complete.

The runtime by default provides a few modules. These are
listed in the machine.config file in the <httpModules>
section. Mono by default includes: FormsAuthentication,
UrlAuthorization, Session and the OutputCache modules. These
hook up to one or more of the stages in the application
pipeline.

These stages are part of the HttpApplication class. You
typically have one of these per "directory" where you have
deployed an ASP.NET application. As a developer, you can hook
up to various points in the processing pipeline. For example,
you could add your own authentication system by listing a
dynamic module in the web.config file, and then hooking up to
the processing pipeline:

app.AuthenticateRequest += my_authenticator;

Now, typically when processing a request the various hooks
are invoked synchronously: one after another. But in some
cases your hook code might want to perform a lengthy
operation, for example contacting a remote authentication
server. Instead of blocking the executing thread, you want
to queue the work using the CIL asynchronous framework and
release the current thread from its duties so it can be used
to process another request. To do this, you must register
your hook using a different API:

Where begin_authentication is the method that will
initiate the authentication asynchronously and
end_authentication is the method that will be invoked
when the asynchronous operation has completed. As I said
before, while the asynchronous operation is pending the thread
should be returned to the threadpool so it can handle the next
request. When the asynchronous operation is completed it will
be queued for execution again and when a thread becomes
available it will execute the completion routine and resume
execution.

The challenge is that you might have both synchronous and
asynchronous hooks in a given application and also that at any
stage the pipeline can be stopped (for example if
authorization failed).

Now the challenge was to come up with a maintainable and
clean design for the application pipeline. Here is where
iterators came into play. The first step to make this simple
was to treat all asynchronous registrations as synchronous at
the event layer:

Notice that we basically are using nested yield-based
enumerators. The return value from "RunHooks" indicates
whether the pipeline must be stopped (true) or not (false).
RunHooks will execute as many synchronous operations
as it can in order until it finds an asynchronous operation.
At that point it initiates the operation calling the "begin"
method and then it yields the control. The control is
transfered to the Pipeline method which also yields and
returns control to the caller.

Execute is basically using the IEnumerator interface
directly while the Pipeline method uses the conveniece
foreach method that iterates over every step.

The complete sample can be obtained here.
This is the prototype I used as a proof of concept. The
actual implementation (with different routine names) that we
landed on System.Web is more complete is available here
(you must scroll down).

At 1000 lines of code for the full file its one of the
clean and small hacks that am most proud of.

Ports: August has been a good month for Mono ports.
Zoltan has completed the JIT port to the IA64 platform,
another 64 bit port of the Mono VM. Wade has made some
RPMs
based on a CVS snapshot that you can install on SLES9/IA64
(Update: Link to an external site with the snapshots).

In the meantime Paolo has made significant progress on the
ARM JIT port. It is now running on both little-endian and
big-endian machines. Geoff even managed to build Mono and run
Mono on his 133 Mhz Linksys
machine

Plenty of progress on the students from Google's Summer of Code:

XBuild: Marek reports that his
implementation of msbuild (xbuild) is now able to compile
itself. Congratulations!

ASP.NET Editor: Michael has merged Blago's code into
the ASP.NET editor and it is now possible to create ASP.NET
controls and also edit the HTML with the Mozilla editor. You
can see his screenshot here.
The code is available from SVN from the module `aspeditor'.

Bug Finder: Aaron report
talks about some real bugs being found by his bug finding
tool. His bug finding tool uses the Cecil library to read the
metadata and analizes the program for known mistakes that
developers incur on.

DIVA: This is probably the flashiest and most visual
of all the summer of code projects. This is a video editor
that uses GStreamer. I recommend you read Michael's blog which is
gives a detailed description of his progress and has various
videos of the various UI widgets he has created to create an
open source video editor with Mono and Gtk#. Michael just
integrated Mono.Cairo into his application to polish various
of his widgets.

Compilers: Florian assisted Cesar in various JScript
tests and the results are very
promising. Mono's JScript compiler is passing more and
more of the Rhino tests. Jaen checked in his
Ruby.NET
compiler into the repository and Florian has been
assisting a bit. Discussion is happening on the #ruby.net
channel on irc.gnome.org.

Work on GCC CIL has made great progress but suffered a set
back as Jey broke his wrist this week.

And finally PHP.NET
is on schedule to complete the goals that Raffa signed up for
in the summer of code. He will continue working on the other
milestones as part of his university work after the summer is
over.

Cecil: Cecil continues to make progress. The best
way of tracking Cecil progress is on JB's blog. It is
now capable of writing assemblies and hopefully soon we can
start on the Mono diet linker.

Monodoc: has received plenty of improvements: now it
uses Mozilla, has font changing features, can summarize
contributions, supports CSS rendering and integrated Lucene
indexing to search the contents of documentation.

ASP.NET: We just landed
a large patch to our implementation of ASP.NET.

Libgdiplus: Now that Cairo 1.0 and the API is frozen
we ported Libgdiplus to use the new version of Cairo. The
code is almost ready to be landed on the main branch but there
are a few loose ends still required before we commit it. The
good news is that Windows.Forms applications are now
significantly faster.

IronPython: Zoltan has fixed pretty much all bugs in
Mono that prevented Mono from running IronPython 0.9 and its
included regression test suite. We have identified a couple
of problems that still must be fixed, but hopefully they will
be in place by the time we release Mono 1.1.9.

MCS Just when we thought that we were done with the
C# compiler Microsoft rectified
a problem in the Nullable Types specification. We are all
glad that they did and Hari is working on fixing this. In the
meantime Marek and Atsushi continue on a quest to eliminate
MCS bugs and to make the compiler stricter and more useful.

There is one regression in the usefulness of mcs that I
have noticed recently. In the past the compiler would
continue reporting errors for other problem areas in your
source, but now it stops too early. Way too early. So
when you have been hacking for a few hours straight you might
need a few builds to get all the errors ironed out. You
always do, but we used to need less.

The problem originates in that in a few places in the
compiler we decided that if we can not compile a chunk it was
best to stop processing and not continue as the compiler might
depend on the types being defined and that might cause the
compiler to crash. At the time, I felt that it was best to
stop the compiler, but now am thinking that maybe we would
serve our users best if we coped with incomplete trees in the
compiler and produce more errors. It wont be easy to change
this though.

Stephen Walli an ex-Microsoftie weights
in on the Mono BOF at the PDC:

They published the CLR and C# language specifications
as international standards through ECMA and ISO, but
it still looks like a proprietary controlled
technology to the customer base. Standards are a
message in the marketplace that encourage multiple
implementations. As long as there is only one
commercial implementation, there is no standard, and
the customers know it regardless of the number of
trees killed to produce specifications.

He goes on saying that we should take our impromptu
gathering beyond the hallway meeting into something larger:

Instead of gathering in a hallway like stubborn
refugees, get a meeting room in a neighbouring hotel,
or take over a bar like a Dick's Last Resort.
[..]
Hand everyone a copy of the latest SuSE
distribution, and whatever Mono tools can be put
together. Hand out t-shirts to the first 100 people
through the door and make them GREAT t-shirts.

Last night's Jon Stewart show kept me on the brink of
suspense since Jon announced that Christopher Hitchens would
be the guest on the program. Hitchens being probably the
best informed person and probably the only articulate person
backing up the war on Iraq is a difficult bone to chew, and I
knew this was going to be an interesting interview.

Jon sometimes chooses to not confront head-on his guests on
the show when they representing dissenting ideas. With
Hitchens it was different, he asked directly "explain to me
why I am wrong" and had the best comments on the debate.

The episode was fantastic, if you missed it Crooks and
Liars has the
interview and in my opinion one of the best openings on talking
points.

My other show

Nat recently introduced me
to Discovery Channel's
Mythbusters a TV show that looks at urban legends and
looks at how plausible those myths really are. Am hooked on
it.

English

Not being a native speaker means that sometimes my spelling
is not great. I depend on Emacs ispell and Evolutions speller
for many things, but recently I have started using Google's
"define:word" to look up word definitions. Its just two
keystrokes away: C-j define:word ENTER.

There are many new things in the new Mono that are worth
showing and talking about to folks at the PDC this year.

Am afraid this year the Mono BOF will not be accepted again
at the Microsoft PDC. My submission was reviewed on Tuesday
and it has still not shown
up in the voting list for BOFs, chances are voting for it will
only happen (if it happens at all) on Saturday/Sunday, clearly
the worst days to get any votes on.

At the last PDC the Mono BOF had the largest number of
votes when half the spots were still available and it got
dropped out of the list. When I asked the various people in
charge what happened they kept pointing fingers at someone
else until it reached full circle. Nobody could tell me why
the most voted BOF proposal did not get selected. I would be
happy with an honest answer even if it is "We do not want to
promote open source/Mono/Novell" instead I heard a number of
variations on "The problem is that `New frontiers for 6502
assembly language in the copy-editing industry had more
votes'" (it didnt).

Anyways, this PDC ts looking just like the last one. So it
is time to get ready for a Mono meeting like we hadthe last
time: in the middle of the hallway. Last time we picked a
spot in the middle of two concourses which had enough space
and chairs to hold our meeting (about 80-100 people). I will
do some scouting on Monday and find a good spot and a good
time to hold the meeting and announce a meeting place here.

If you get to the PDC early, and you know the spot where we
had the meeting the last time and you can provide directions,
please email me.

Please only request to be added if we have IMed, talked in
person or emailed each other in the past.

Instructions for Gnome/Gaim users: here.
Google's IM is based on the open Jabber protocol which not
only allows for many implementations and creative uses of IM,
but in addition Jabber is an interesting protocol that can be
used to route arbitrary XML messages across the internet (not
necessarily IM only).

The protocol has now a big service provider with a big user
base that will help realize its potential.

Once you let Iraqis buy
your food on the streets, tell you what people are saying,
come back to you with their observations, you have entered the
pointless hothouse of hotel journalism, the reporter with the
mobile phone trapped in his room who might as well be
broadcasting or writing from Co Mayo.

His Pity the Nation book has many colorful stories of hotel
journalism as practiced by many in the days of the Lebanon
civil war: from the bar at the Commodore Hotel when things got
risky to reporting on the crisis from the nearby Island of
Cyprus.

There is a new twist to "hotel journalism" as things go
from bad to worse in Iraq. The Lebanon years of
Hotel-journalism are gone, replaced now with Prison-journalism:

I head off to the Palestine Hotel where one of the largest
Western news agencies has its headquarters. I take the lift to
an upper floor only to be met by a guard and a vast steel wall
which blocks off the hotel corridor. He searches me, sends in
my card and after a few minutes an Iraqi guard stares at me
through a grille and opens an iron door.

I enter to find another vast steel wall in front of
me. Once he has clanged the outer door shut, the inner door is
opened and I am in the grotty old hotel corridor.

The reporters are sitting in a fuggy room with a small
window from which they can see the Tigris river. One of the
American staff admits he has not been outside "for months". An
Arab reporter does their street reporting; an American travels
around Iraq - but only as an "embed" with US troops. No
American journalists from this bureau travel the streets of
Baghdad. This is not hotel journalism, as I once described
it. This is prison journalism.

He has been reporting back from the streets of
Baghdad since August 12. Where accidents and death are
treated in a way similar to Terry Gillian's Brazil's
futuristic state: fill
form completely.

In all - and this was only an initial count - 43 civilians
were killed and more than 80 wounded in the deadliest bombing
in Baghdad this month.
[..]
For once, it seemed, there were no suicide bombers
involved, just old-fashioned car bombs, packed with explosives
to kill the largest number of innocents in the least possible
time.

"I consider this a quiet day," one of the mortuary officials
said to me as we stood close to the dead. So in just 36 hours
- from dawn on Sunday to midday on Monday, 62 Baghdad
civilians had been killed. No Western official, no Iraqi
government minister, no civil servant, no press release from
the authorities, no newspaper, mentioned this terrible
statistic. The dead of Iraq - as they have from the beginning
of our illegal invasion - were simply written out of the
script. Officially they do not exist.

Thus there has been no disclosure of the fact that in July
2003 - three months after the invasion - 700 corpses were
brought to the mortuary in Baghdad. In July of 2004, this rose
to around 800. The mortuary records the violent death toll for
June of this year as 879 - 764 of them male, 115 female. Of
the men, 480 had been killed by firearms, along with 25 of the
women. By comparison, equivalent figures for July 1997, 1998
and 1999 were all below 200.

Fisk quotes an Iraqi on what seems
obvious to anyone but the Chenney administration:

As for the constitution, I asked an old Iraqi friend what he
thought yesterday. "Sure, it'ss important," he said. "But my
family lives in fear of kidnapping, I'm too afraid to tell my
father I work for journalists, and we only have one hour in
six of electricity and we can't even keep our food from going
bad in the fridge. Federalism? You can't eat federalism and
you can't use it to fuel your car and it doesn't make my
fridge work."

And then someone told me the other day, and am not kidding
you "at least they have democracy now". Fisk listens to a
military commander weight on
the bus station bombing:

And that
night, I flip on the television again and find the local US
military commander in the Sadr City district of Baghdad -
close to the bus station - remarking blithely that while local
people had been very angry, they supported the local
"security" forces (ie the Americans) and were giving them more
help than ever and that we were - wait for it - "on the path
to democracy".

You cant make stuff up like this.

I started blogging today about Utah's
events. But they now seem minor issues.

Last week I broke my elbow when I fell off my bike. The
humerous "tip" got broken and required surgery and three large
screws to hold it in place. I wish at least I had broken it
as part of a facinating story, involving a remote country,
spies and a distress call. But all I got is "I was coming
back from work and fell".

I have tried to reconstruct the tragic events of 8/12 to
find the failure on the system. But I have come up empty
handed so far.

What I do know is that in the morning I used Nat's air pump
on my bike and got a kick out of it. I remember thinking "I
think I over inflated these bad boys".

That afternoon after a frantic hacking expedition removing
"throw NotImplementedExceptions" from our new code and filing
some compiler bugs, I realized I only had 20 minutes left to
meet my wife on bthe opposite side of the river.

The lock holder had loosened up, I took the lock out and
put it on my back pack.

Within 10 meters of getting on the bike I landed
emarasingly in front of a bunch of guests of the Residence
Inn, the hotel next door to the office. Being the proud man
that I am, I picked the bike and got moving right away, took
me a while to realize that I could not move my left arm.

My doctors have been awesome. My only complain is that I
had to wait six days to get my surgery done and on the ER it
took them the best part of four hours to give me a pain
killer.

I started right away my rehabilitation. Took me almost two
days to get my hand to lift again, and my fingers are moving
now, but they cant really exert any pressure. My nose has
more strength. I also painfully learned that you are supposed
to keep your affected limbs above your hearth, so even if the
fingers can move (and I managed to type a paragraph the other
day) I cant dfind a position that wont make my hand swell.

This comes at a very bad time: this is the third year on a
row that I have missed FooCamp; The Google sponsored Mono
students are progressing by leaps and
bounds on their projects, Mono is getting ported to twonew
platforms, Minosse was
completed. and my hacking has stopped for the time being. In
particular, am very proud of a new htp pipeline design that we
are about to land in Mono which uses iterators extensively to
maximize the use of the threadpool. Luckly it got done before
the accident.

Reading books has also slown down, its very uncomfortable
so am doing most of my reading from the computer. I almost
finish reading all the web.

The new version of Gtk+, a library used to build
GUI applications on Unix and the heart of the Gnome desktop
ñwas announced
this weekend.

This is a major upgrade for developers creating GUI
applications for Linux and Gnome as it makes it simpler to
create visually reach applications. This new version of Gtk+
is built on top of the 2D graphics engine Cairo: every widget is
now written using Cairo operations and most importantly
developers can now draw their own widgets using the PDF-like
rendering model offered by Cairo.

Historically, the most painful part of building Gnome
applications was the drawing layer: any interesting
visualization or custom widget that needed to draw had to deal
with some fairly low level APIs designed in the 80's: visuals,
colormaps, GCs, collor allocation, resource acquisition and
release (brushes, color resources, pens, patterns). With
Cairo all of these problems will become history.

In addition to the productivity boost and the innovative
potential to create visually rich applications, Cairo also
brings to the end user nice touches like anti-aliased
rendering for a more pleasant experience. Gtk builds on
this new functionality to bring vector-based themes to the
desktop as well.

Atsushi Enomoto has a great list of hints on developers
using XML and Mono's XML if they care about performance, his
list is available here.

String Collation

Mono originally was using IBM's Classes for Unicode (ICU)
library. A C library that provides many tools to handle
internationalized strings (like comparing strings, finding
substrings, handling case sensitivity in a culture-aware
fashion) and so on.

Basically all the managed code in Mono would call into the
C runtime and the C runtime would use ICU's functionality to
carry out the job. Unluckly Microsoft's behavior of the
unicode operations differed from ICU implementation and the
fixes that we applied in our wrapper code that use ICU were
insufficient to provide the same semantics. Developers were
running into various unexpected problems and erratic behavior
that came out of our mapping which prompted us first
discourage the use of ICU, and later to completely disable the
ICU support code in Mono.

A few months ago, Atsushi was wrapping up his work on
System.XML 2.x and asked me what should he look into as his
next task. I asked Atsushi to look into implementing a
replacement for ICU that we could use for Mono. He took this
challenge very seriously and this past week he finally landed
the new string collation code in the repository.

His latest
blog post has some performance information and he links to
the various posts that detail his quest into implementing
string collation for Mono.