[p2k9] Ports Hackathon - Budapest, Hungary

Contributed by
jcroberts
on 2010-03-16
from the slacking-undeadly-editors dept.

Although more than a bit late getting posted here, you should know about the
accomplishments of the p2k9 Ports Hackathon.
Sure, it might look like a party, but when you get a bunch of people
together with similar ideas of having fun porting programs, amazing amounts
of supposed "work" gets done.

I arrived in the building of the hackroom on the Friday afternoon. I
walked around the building blindly for a while, ending up in some data
centre, where bearded UNIXy looking people were speaking at me in
Hungarian (probably wanting to know what I was doing in their data
centre).

After finding Robert, I set up in the hackroom and started working on
some ports I had hanging around in my tree as well as some new stuff:

Some of this I did not fully complete. The TeX Live work was based upon
a nightly snap, as it has not yet been released. The good news is that
it requires far fewer patches than before and should be a bit easier to
integrate.

Myself and espie@ are still working in QT-creator, as it has various
issues. Hopefully we can nail that soon.

On the Sunday naddy@ created my CVS account! My first commit
was shortly after. (Thanks naddy!)

In the evenings we usually went to a restaurant for food and beer. Early
in the week we had a great curry, then later in the week we went to the
legendary Sir Lancelot's for a medieval feast. One afternoon we
slacked at the spa for a few hours and got wrinkly.

The hackathon was both productive and fun. I would like to take the
opportunity to thank the project for allowing the event to happen,
Robert for planning everything and also the other porters for having me
there. Thanks

Matthias Kilian (kili@)

Update lang/guile
and depending ports, mainly to be able to update
print/lilypond.
After some initial attempts earlier this year, where
okan@ and me started to test-build all the depending ports and
looking at the results of make regress, jasper@ helped a lot some
weeks ago by putting the diffs into a bulk build. I didn't want
to commit it before p2k9, though, because of possible breakage.
Only at an event like this hackathon you're able to find and fix
problems quickly.

Update print/poppler,
and help edd@ to let his latest work on texlive
to build with it (it did build with poppler-0.10, but poppler-0.12
broke it, so I fixed it).

After edd@ committed the update of xpdf, I started to merge
the patches into ports based on or including xpdf sources, like
kdegraphics (x11/kde/graphics3),
texlive (the version currently on ports) and poppler.
Some older xpdf changes went also in. The texlive aren't committed
yet, and I'm also working on merging the patches into the stable
branch.

The `traditional'
print/ghostscript
update (only sent to ports@ but not
yet committed, because I want to get people a chance to test it).

Some work on updating
net/kismet
to the latest version. This is still work
in progress, but I think it will be finished early enough for 4.7.

Last but not least, I did some work on getting
lang/ghc
back into shape.
It's still not bootstrappable from scratch, but I think I got a
little bit closer.

Well, no thrilling non-ports stories from me, except you want to
read a big lament about travelling from Braunschweig to Budapest
and back by train ;-)

I was only able to be at p2k9 for four days but it was great to meet everyone
and see all the ports stuff going on and, after some initial setup
tribulations, it was very productive - I was able to drastically reduce the
number of mails in my inbox and diffs lying around in source trees.

I was working on tmux,
mainly on cleaning up some longstanding bugs, queued
diffs or todo items rather than adding any huge features. Of the 39 tmux
commits, these were the most significant:

Added simple job scheduling so that when you run tmux itself from inside
(such as from #() in status-left) it doesn't hang and so that commands are
run only every status-interval rather than every redraw.

Finished off and committed a diff to add "grouped sessions", allowing a
session to share windows but have an independent current window and options,
this is similar behaviour to "screen -x" and has been much requested.

Stop tmux outputting a hard newline at the end of every line even if it
wrapped at the screen edge. Instead, let the terminal wrap naturally,
allowing xterm mouse selection to correctly find the end of the line. This
was another MUCH requested fix, but was quite difficult and frustrating to
get right - without p2k9 I probably wouldn't have been able to spend the time
it needed.

A "mouse-select-pane" option to allow the active pane to be chosen by
clicking on it with the mouse. Suggested before by someone but brought up
again at p2k9 by sthen@.

Committed the diff for a "pipe-pane" command to send any output in a pane
to a command. Requested by espie@.

Scroll lines into the history even when the scroll region doesn't cover the
whole screen, this makes ircII users like naddy@ happy.

Permit attributes to be turned off in #[] by prefixing "no", such as
"noblink".

Use relative cursor movement when the terminal supports it. This makes things
a bit more efficient, and shook out a number of redraw bugs.

Committed a diff a user submitted extending the lock timer to work
independently for individual sessions.

Lots of code cleanup, bug hunting, and some work on making tmux and vttest
play more nicely together. And beer.

I also sent off some minor fixes for some other areas, made a couple of trivial
ports changes.

Due to the bad weather we were pretty much limited to hacking and
eating. But we did discover this AMAZING Swedish Eurodance artist
Gunther and his Sunshine Girls...

Reminded by some recent questions on misc@, I finally sat down and
fixed luit(1),
which is not a port but part of our X11 installation.
By itself, xterm(1)
only supports the ISO 8859-1 and UTF-8 character
encodings. For other encodings required to write many languages,
such as ISO 8859-2 for Czech or KOI8-R for Russian, xterm will run
in UTF-8 mode and spawn luit(1) as a wrapper that translates from
and to the respective encoding. luit lacked proper support to
allocate a pseudo-TTY on OpenBSD and would barely run, issuing a
scary message: "Warning: could not change ownership of tty -- pty
is insecure!". I fixed this by adding proper support for
openpty(3),
so you can now safely use xterm with a large number of traditional
character encodings.

If you had to learn French as a foreign language, you are painfully
aware that there are a few hundred irregular verbs with unpredicatble
stem changes in the many inflectional forms. The new
education/verbiste
port can demonstrate the conjugation of even the most
tricky French verbs. And if you come across a strange verb form
in a text, verbiste can also tell you which tense/mood/person of
which verb(s) you are looking at, so you know what to look up in a
dictionary.

Larger port updates I committed at p2k9 were
audio/mpeg123,
which
now uses optimized assembly code on more platforms, and in particular
graphics/jpeg v7.
A lot of ports directly or indirectly depend on
jpeg, so I needed to run a full bulk build, but happily only two
ports required special fixes.

phessler@ suggested that it would be nice if
audio/openal,
a 3D
audio API used by a number of games, supported our native sndio
audio interface instead of having to resort to OSS emulation. As
usual, there was no documentation on the internal API, so I had to
spend a silly amount of time reading the backends for other operating
systems before I felt confident to write what turned out to be a
surprisingly simple sndio backend. This still requires some testing
before it can be committed.

And, as usual, I also took the opportunity to commit a number of
minor maintenance updates and bug fixes too boring to list in detail.

There was a student from 1st year Epitech, Florent Tribouilloy, whom I
invited, and who did tasks such as helping Antoine on a
editors/subtitleeditor,
update and Landry on some
x11/xfce4/terminal,
update. He's been learning quite
a few ropes of the ports tree. Before that, I assigned him to work on some
internal school project: write a specific bsd.rd that can pass their
rather unusual proxy system (which he did) so that people at epita/epitech
can install OpenBSD over the network without having first to download all
the sets using another OS. I've also asked him to have a look at writing a
BSD-licenced
devel/cflow
based on the lint parser... no need to reinvent the
wheel, since we already have a C language parser in the tree. This might
help in finding horrible code in lint, to boot ;-)

As far as I'm concerned, I mostly did work on
pkg_add.
Most of the changes
don't have a lot of user impact, so far. I'm cleaning up the code to allow
for two things:

merging the update system with the dependency solver. So that updates will
eventually happen "on the fly", instead of computing all updates first, and
doing them later. This allows me to be more correct in the dependency order,
also to do partial updates correctly, and to install new packages while
updating their dependencies. If you're in a hurry, this means you will be able
to start an update and have it do stuff quickly, instead of waiting forever
until it has scanned your whole list of packages. I've had this in my tree
for ages, but there were bugs to fix. I had to clean up dependency handling
an to add a new "tracker" object that knows about all pending updates just
so that the code doesn't get any messier.

merged updatesets. That is, updating two or three packages at once, in cases
where it's necessary: tight dependencies (mysql-client/server), files moving
from package to package (kdelibs/kdebase), or even dependency reversal
(somewhere in cairo/pango a few releases back). Most of the infrastructure is
now there. What's left to do is to recognize those situations, and check
that an updateset with newer > 1 works. This is fairly important for
correctness, since some update scenarios currently do cheat and work by
accident.

-z/-l option, to duplicate an install (more or less) quickly. Best effort
so far. Will work much better once the "update-on-the-fly" thingy works.

cosmetic clean-ups of install messages: less lines during updates. Restoring
dependency handling. Showing signed packages in a cleaner way.

recognizing some cases where repositories don't have packages (wrong path)
that we didn't get so far.

I also worked on ports a bit, mostly with Edd, who's trying to make progress
on cmake (which is a tough issue), and on qt-creator, for which you should
expect a port sometime in the near future: the actual issue is just a question
of path to the plugins, which is recorded nowhere except on linux.

There was also a pending patch from Nicm that I validated over p2k9: the
pipe-pane command, which can be used with a variation of portslogger to log
all activity in a tmux window that's related to the ports tree. Very cool.

I was happy to meet edd, phessler, and nicm. And of course, to see again the
usual band of suspects.

Regarding ports, i imported some standalone stuff like
audio/cmixer,
a quick hack written in perl-curses by yours truly to display a console mixer, or
games/jbrickshooter,
a stupid reflexion game which made me dreaming of moving
coloured blocks, ajacoutot@ can witness it's addictive (and stupid).. i fixed some
x11/gtk+2,
update breakage, mainly in
www/kazehakaze,
(which required some
awful hackery with axe and rocketlaunchers, some parts of this code are
really horrible and made me produce a lot of grumblings and rants).

I also updated
www/webkit,
to a more recent version, now that we have a recent
devel/libsoup,
and updated various
x11/xfce,
components (that's a background task).
And the two most important items :

This required working on a lot of
dependencies : i took some from djm@'s work on
multimedia/coherence
and
audio/py-tagpy,
and worked on
net/libmms
support in gstreamer.
devel/py-cssutils
gave me some work too, but the
hardest part was definitely
graphics/pigment
and
graphics/py-pigment,
which required some help from oga, guenther and kurt to understand how
py-pigment & python were supposed to load and use libGL. I spent two days on it, but
finally figured out a "clever" workaround, and was able to successfully
run it on my atom netbook with decent performance.

I had a quick (one night) look too at
XBMC,
another HTPC/Media player software,
and there's a huge amount of work before having this beast running on
OpenBSD... maybe someday; it's a huge pile (160Mb tarball) of awful code
from various sources, linking tons of homemade patched copies of
ffmpeg, libdvd* & various video libs, an embedded ms-windows dll virtual loader,
a homemade VFS layer, and tons of linux/osx/windows-only functions. A good
looking app for sure, but not what i'd call portable to OpenBSD... a
make clean in the wrong dir ruined my patching efforts, so it's pushed for
another day :)

And most importantly, i enjoyed sharing some good french & dutch cheese and
homemade mirabelle with the porters crew, eating to death at sir
lancelot, trofea or in other really awsum' restaurants serving crazy liver-based
meals.. that was a really cool edition, way too short !

work on bringing
math/octave
up to date (this is still WIP but all
the major parts were done during c2k9)

do some OT stuff like ACPI suspend/resume work and
investigation (a big patch was commited during p2k9 actually)

bring in some new ports like pdq and other ruby utilities

plus lots of small stuff that I don't remember at the moment

FUN: (i.e. Other FUN)

drink beer and toss ideas at night

revisit pubs from the last hackathon

cool restaurants like the indian one we went to last year

Sir Lancelot

Spa (I didn't attend, but others went)

I spent the weekend after p2k9 with my girlfriend visiting the
old center and some of the other interesting parts of Budapest,
going to the spa, going to the dead baby heads pub and eating at some of
the nice restaurants I found during the hackathon

I spent most time on
devel/libtool.
Code cleaning, and separation into smaller,
more understandable pieces, as well as fixing a few corner cases.
Right after p2k9, I was happy to see that it can now finally complete a
build of xenocara. It is also able to build some more other ports correctly.
I need to do some more testing with it, and if the result is good enough,
we can enable it for general use.

The new libtool allows us to more easily build programs or libraries even
when other versions are installed on the system, and will also speed up
ports bulk builds (about 20% of the ports tree uses libtool, many of which
are big ports), so it helps to reduce the time needed to build a package
snapshot.

I also spent some time helping pirofti@ with getting
math/octave
to build;
the gfortran compiler and associated fortran libraries are needed there,
and this required some fixes in the
lang/gcc/4.2
packages.
Some more minor things, helping out new people a bit, and the
activities... I think we finished the plate this time :)

p2k9 had a pretty clear goal for Antoine and myself; update our
x11/gnome
ports to their 2.28.x release.

Pretty early on I made the "mistake" of queuing some fine Swedish cultural
gems...and pretty soon thereafter everyone was familiar with Guenther's tralala
and ding-ding-dong. Oh, and having a beamer in the hacking room didn't really
help either.

Some time before p2k9 major updates to
devel/glib2
and
x11/gtk+2
had already been
committed, which allowed us to really focus on GNOME. Instead of having to worry
about those important libraries too. We set off right from the beginning at
updating and pretty soon we ran into "old friends". Being HAL, PolicyKit and
GStreamer. Stuart is doing some great groundwork which will eventually allow us
to update GStreamer. He'll elaborate on that though.

As it turned out, even with two people hacking on GNOME for one week, almost 24
hours a day, we still weren't able to finish all the things we wanted to. There
are some regressions that we noticed, but we continue to work on/debug it.

Five of us had decided to go for some serious and well deserved slacking after
p2k9, and on Friday we headed to Hévíz. Where we slept, swam and drank to
prevent post-hackathon burn-out...

As was
written on ports@
we would appreciate your testing and feedback! And thanks
again to Robert for organizing yet another superb edition of the ports hackathon
in Budapest.

Stuart Henderson (sthen@)

As well as updating some ports I maintain (and a few which I'll
need in order to update some others sometime in the future),
I spent some time working with a newer version of
devel/flex
which is
needed in order to update various ports (including gstreamer, which
itself blocks many other port updates).

My initial aim was to update
the version in base and started on that basis (beinning with a diff
from millert@ and working towards updating the manual from the info
pages) while running a bulk package build. But after looking over
the failure logs, it doesn't seem quite ready to make it the only
version in the system, so I've imported it as a port for now.

And of course we had the usual mix of awesome music,
and a really nice time in Budapest (and then a few of us
stayed the following weekend near the lake in Hévíz for
some much-needed recovery).

Due to slacking on the part of the undeadly editors, the article lags
behind reality a bit, so some of the changes described here have been committed a
long time ago. None the less, it's always nice to see work and fun behind
the `sudo pkg_add whatever` magic.

NOTE: I am the one responsible for putting embedded youtube files on undeadly,
and there have been more than a few questions about it, particularly regarding
Adobe/MacroMedia shockwave. First of all, the actual videos are a running
joke with some of the devs, and even the act of embedding youtube videos on
undeadly is also a bit of a joke, but most importantly, you should already
know how to handle video sites and formats with OpenBSD. If not, then send me
hate mail until I write an article explaining how to deal with it.

Now please excuse me while I go watch sthen's lattest addition, and then
promptly go running off into the weeds, screaming.

(Comments are closed)

By
Timo Myyrä (zmyrgel) timo.myyra@gmail.com
on 2010-03-16 13:18

Great article as usual.

It's good to see that lang/ghc has not been forgotten.

By
Matthias Kilian (kili)
on 2010-03-16 20:55

> Great article as usual.

Well, it's a little bit embarrasing if you see your promises for 4.7 that you didn't done in time ;-)

> It's good to see that lang/ghc has not been forgotten.

Of course not. I tried to get GHC boostrapping work since it has been broken. But ir's incredibly frustrating; whenever you get some small step done (like `stage1 finally compiles from .hc files) you run into the next problem (like `but the linker fails'), and have to dive into the still overcomplicated build system and into its dependency hell.

But after I still didn't manage to even get a working ghc bootstrapped from .hc files on the very same platform I created the .hc files, I know decided to join the dark side of the power[tm]. I recently started to prepare "minimal" precompiled binaries (for i386 and amd64) that then can be used to run the main GHC build. Of course, testing wether those minimal precompiled binaries (with debugging and profiling libraries as well as GHCi removed) are enough to start a full GHC build revealed, that it does not yet work. Surprise! Some library currently included in GHC (dph) can't be built without GHCi available. After my usual rant on the GHC lists, I'm now running some builds with dph removed from GHC (since dph isn't required at all by GHC), and I hope to get a patch ready for upstream.

Anyway, switching to the dark side of the power means that lang/ghc is stuck to i386 amd amd64 forever. .hc bootstrapping didn't work for more than three years and is a dead-end (IMHO), crosscompiling still doesn't work (and even if it worked, I wouldn't have the time nor the resources to port the rts and maintain binary builds for all OpenBSD architectures), and there's no other Haskell compiler that can be used to build GHC.