There is a bad trend going on that seems to purport Gentoo’s QA (which for the past couple of years has meant mostly me alone), as a perfectionist hindering getting stuff done — and I call all of this utter bull feces. It’s probably telling that the people who seem to then expect that every single issue has an explicit written rule, with no leeway for different situations.

So let me give you some insights so that you can actually get a realistic clue of what’s going on. Rich is complaining that I made it a task of mine to make sure that the software in Portage don’t use bundled libraries; for some reasons, he seems to assume that I have no clue on how cumbersome it is to deal with said bundled libraries, and he couldn’t be more wrong. You know what my first bundled libraries project? xine. And it took me quite a long time to get rid of (almost all) the bundled libraries; even more so because many of said libraries were more or less heavily modified internally. Was it worth it? Totally. It actually made the xine package much more reliable to security issues (and there had been quite a few), as well as solving a number of other issues, including the most infamous problem with the symbols’ collisions between the system’s libfaad (brought in by ffmpeg) and the internal copy of it.

So, I know it’s a lot of work, and I know it’s not always a task for the faint of heart, and most of the times there is no immediate improvement by fixing those. Why am I insisting on it as a point of policy? Because we have quite a few examples with software being bundled for which vulnerabilities were found and could be leveraged by an attacker, especially for what concern zlib and software that either downloads or receives compressed content over the network.

So from what Rich wrote, we’re trying to hinder getting stuff in tree by refusing it if it bundles libraries. That’s so much of a lie that it’s not even funny. We have had packages entering the tree with bundled libraries, and I’m pretty sure there still is a ton of it. Heck, do you remember what I wrote about Blender just last month? Blender has a number of bundled libraries, and yet it’s in tree, I maintain it, and it’s going stable from time to time.

What is important in that scene? That I’m trying to get rid of said libraries. The same applies to Chromium and most other packages that have a ton of bundled libraries; most packages are responsible enough, and generally know enough about the package that they can work on getting rid of said libraries, if that’s feasible at all — in the case of Chromium it’s an extremely difficult task I’m sure, mostly because upstream does not care the least, and that was clear at last VDD when we were discussing the patches applied over ffmpeg/libav.

So let’s get into the specific details for the complains, as Rich’s “just an example” is not an example of what happens at all. There is a server software written by Logitech for their SqueezeBox devices, that is now called logitechmediaserver-bin in Gentoo. Said package has been known to be a bundled for years — Logitech is bundling a large set of CPAN modules with it, as it seems like the bulk of the code is written in Perl or something like that. I’ve known it at quite a few versions to bundle a CPAN module that, in turn, bundled an old copy of zlib, one that is vulnerable to a few different issues. Right now, it’s not only bundling a bunch of modules, but I found out that it installs a number of files for architectures that are completely incompatible with your system (i.e. a PowerPC binary on amd64). This bothered me quite a lot more. Why? Because it means that the (proxy) maintainer is not discriminating at all, and just installing whatever the upstream package comes with. Guess what? That’s not a good way to package the software.

When the proxy maintainer is not doing stuff that gets even nearby the quality level of most of the rest of the tree, and the Gentoo developer who should be proxying him ignoring the problems, things are messy enough. But it gets worse when you get a person known for bad patches sold as fixes from over three years ago, who still expect to be able to call the shots.

If you see bug #251494 he’s actually suggesting marking the package stable because it’s not going to run with the current unstable Perl which is a completely backward reason (if it can’t work with the latest Perl, it means that it hasn’t been tested in testing for a long time), and is going to create more trouble later on (the moment when the new Perl goes stable, we have a known broken, unfixable package in stable). But he’s Daniel Robbins, and some people expect that that’s enough for him to be right — if that was the case, I suppose he’d still be the Gentoo Linux lead, and not just of a wannabe project.

Anyway, here’s the deal: QA policies are more like a guideline, in general. On the other hand, there is no reason why we should be forced to mark things stable if they do not follow the common QA policies — especially for proprietary software, and software requiring particular hardware, marking things stable is not that great an idea, as they tend to be orphaned quite easily if a single developer retires. We already have quite enough packages stable in x86, ppc and sparc that are not really able to run, because they were always broken, but has been stabled in ancient times. Sometimes, we even have packages keyworded that cannot be used on a given arch, but they do build, and the failure would only happen at runtime, so they have been, again in ancient times, keyworded.

Maybe this is what people who want to follow Daniel expect: coming back to the “it builds, ship it!” mentality that made Gentoo a joke in the eyes of almost everybody at the time. For sure, it’s not what I want, and I don’t think it’s what the users as a whole group want or need.

For ages, I've wanted an option in Skype or Empathy to record my video and voice calls1. Text is logged constantly because it doesn't cost much in the form of resources, but voice and video are harder.

In lieu of integrated support inside Empathy, and also because I mostly use Skype (for various reasons), the workaround I have is to do an X11 screen grab and encode it to a file. This is not hard at all. A cursory glance at the man page of avconv will tell you how to do it:

avconv -s:v [screen-size] -f x11grab -i "$DISPLAY" output_file.mkv

[screen-size] is in the form of 1366x768 (Width x Height), etc, and you can extend this to record audio by passing the -f pulse -i default flags to avconv2—but that's not quite right, is it? Those flags will only record your own voice! You want to record both your own voice and the voices of the people you're talking to. As far as I know, avconv cannot record from multiple audio sources, and hence we must use Pulseaudio to combine all the voices into a single audio source!

As a side note, I really love Pulseaudio for the very flexible way in which you can manipulate audio streams. I'm baffled by the prevailing sense of dislike that people have towards it! The level of script-level control you get with Pulseaudio is unparallelled compared to any other general-purpose audio server3. One would expect geeks to like such a tool—especially since all the old bugs with it are now fixed.

So, the aim is to take my voice coming in through the microphone, and the voices of everyone else coming out of my speakers, and mix them into one audio stream which can be passed to avconv, and encoded into the video file. In technical terms, the voice coming in from the microphone is exposed as an audio source, and the audio for the speakers is going to an audio sink. Pulseaudio allows applications to listen to the audio going into a sink through a monitor source. So in effect, every sink also has a source attached to it. This will be very useful in just a minute.

The work now boils down to combining two sources together into one single source for avconv. Now, apparently, there's a Pulseaudio module to combine sinks but there isn't any in-built module to combine sources. So we route both the sources to a module-null-sink, and then monitor it! That's it.

1. It goes without saying that doing so is a breach of the general expectation of privacy, and must be done with the consent of all parties involved. In some countries, not getting consent may even be illegal.2. If you don't use Pulseaudio, see the man page of avconv for other options, and stop reading now. The cool stuff requires Pulseaudio. :)3. I don't count JACK as a general-purpose audio system. It's specialized for a unique pro-audio use case.

Voltaire once said that the best is the enemy of the good. I think that there are few places where one can see as many abuses of quality as you’ll find in many FOSS projects, including Gentoo.

Often FOSS errs on the side of insufficient quality. Developers who are scratching itches don’t always have incentive to polish their work, and as a result many FOSS projects result in a sub-optimal user experience. In these cases “good enough” is standing in the way of “the best.”

However, I’d like to briefly comment on an opposite situation, where “the best” stands in the way of “good enough.” As an illustrative example, consider the excellent practice of removing bundled libraries from upstream projects. I won’t go on about why this is a good thing – others have already done so more extensively. And make no mistake – I agree that this is a good thing, the following notwithstanding.

The problem comes when things like bundled libraries become a reason to not package software at all. Two examples I’m aware of where this has happened recently are media-sound/logitechmediaserver-bin and media-gfx/darktable. In the former there is a push to remove the package due to the inclusion of bundled libraries. In the latter the current version is lagging somewhat because while upstream actually created an ebuild, it bundles libraries. Another example is www-client/chromium, which still bundles libraries despite a very impressive campaign by the chromium team to remove them.

The usual argument for banning packages containing bundled libraries is that they can contain security problems. However, I think this is misleading at best. If upstream bundles zlib in their package we cry about potential security bugs (and rightly so), however, if upstream simply writes their own compression functions and includes them in the code, we don’t bat an eyelash, even though this is more likely to cause security problems. The only reason we can complain about zlib is BECAUSE it is extensively audited, making it easy to spot the security problems. We’re not reacting to the severity of problems, but only to the detectablity of them.

Security is a very important aspect of quality, but any reasonable treatment of security has to consider the threat model. While software that bundles a library is rightfully considered “lower” in quality than one that does not, what matters more is whether this is a quality difference that is meaningful to end users, and what their alternatives are. If the alternative for the user is to just install the same software with the same issues, but from an even lower quality source with no commitment to security updates, then removing a package from Gentoo actually increases the risks to our users. This is not unlike the situation that exists with SSL, where an unencrypted connection is presented to the user as being more secure than an SSL connection with a self-signed certificate, when this is not true at all. If somebody uses darktable to process photos that they take, then they’re probably not concerned with a potential buffer overflow in a bundled version of dcraw. If the another user operated a service that accepted files from strangers on the internet, then they might be more concerned.

What is the solution?: A policy that gives users reasonably secure software from a reputable source, with clear disclosure. We should encourage devs to unbundle libraries, consider bugs pointing out bundled libraries valid, accept patches to unbundle libraries when they are available, and add an elog notice to packages containing bundled libraries in the interest of disclosure. Packages with known security vulnerabilities would be subject to the existing security policy. However, developers would still be free to place packages in the tree that contain bundled libraries, unmasked, and they could be stabilized. Good enough for upstream should be good enough for Gentoo (again, baring specific known vulnerabilities), but that won’t stop us from improving further.

It has been a while since I have last written here but I am not dead and I still somehow manage to contribute to Gentoo.

In the past weeks, I have been working on making Gnome 3.6 ready for inclusion in portage. It rapidly appeared that Gnome 3.6 would have to use both gstreamer 0.10 and gstreamer 1.0 however gstreamer team is badly understaffed and only Alexandre (tetromino) who is not even a gstreamer team member had tried to start bumping ebuilds to gstreamer 1.0.

But then Alexandre got busy and this development stalled a bit. After I finished bumping the overlay to Gnome 3.6.1, I took the challenge to rewrite the gstreamer eclasses to make them easier to use and understand. They were, in my opinion, quite scary with version checks everywhere and I think it is one of the reason that so few people wants to work in gstreamer team

If you do not follow gentoo-dev, most of the code moved to gst-plugins10.eclass which received some magic touches that basically makes 99% of the version dependant code go away. As an added bonus, the eclasses are now documented and support EAPI 1 to 5. EAPI 0 support got dropped because of missing slot operators which is really annoying right now with gstreamer.

So if you hit some gstreamer compilation problems in the last few days, please forgive me, upgrade road was a bit bumpy but, overall, it was not so bad. And now, I am happy to say that gstreamer 1.0 is in portage which clears the road for gnome 3.6 inclusion.

On a final note, I also continued Alexandre’s work of bumping last 0.10 releases and so we are up-to-date on that front as well.

I’ve noticed that not that many resources are online telling you how you can use nginx as a reverse SMTP proxy. Using a reverse SMTP proxy makes sense even if you have just one mail server back-end, either because you can easily switch towards another one, or because you want to put additional checks before handing off the mail to the back-end.

In the below example, a back-end mail server is running on localhost (in my case it’s a Postfix back-end, but that doesn’t matter). Mails received by Nginx will be forwarded to this server.

If you first look at the mail setting, you notice that I include an auth_http directive. This is needed by Nginx as it will consult this back-end service on what to do with the mail (the moment that it receives the recipient information). The URL I use is arbitrarily chosen here, as I don’t really run a PHP service in the background (yet).

In the http section, I create the same resource that the mails’ auth_http wants to connect to. I then declare the two return headers that Nginx needs (Auth-Server and Auth-Port) with the back-end information (127.0.0.1:25). If I ever need to do load balancing or other tricks, I’ll write up a simple PHP script and serve it from PHP-FPM or so.

Next on the list is to enable SSL (not difficult) with client authentication (which isn’t supported by Nginx for the mail module (yet) sadly, so I’ll need to look at a different approach for that).

BTW, this is all on a simple Gentoo Hardened with SELinux enabled. The following booleans were set to true: nginx_enable_http_server, nginx_enable_smtp_server and nginx_can_network_connect_http.

a last-minute contribution to this junto. i was in a car wreck a couple days ago, so abruptly my planned participation time was reduced to just a day and a half. i could only spend a little while per session sitting at the DAW. the track’s title is a reference to that event.

the three source tracks were very hard to work with; this was easily the hardest junto i’ve attempted. i had to make several passes through the tracks, pulling out tiny sub-one-second sections here and there, building up percussion, or finding a droney passages that would work for background material.

for the percussion, i zoomed in and grabbed pieces of non-tonal audio, gated them to remove incidental noise, and checked playback at other speeds for useful sounds. some of the samples were doubly useful with different filter and speed settings. most of the percussion sounds were created after isolating one channel or mixing down to mono; this gave a sharper, clickier sound. occasionally, some of the hits/sticks were left in stereo for a slightly fuller sound.

the melody/drone passages were all pulled from the “unloop” track. i chopped out a short section of mostly percussion-free sound at the beginning of the song, isolated one channel, and ran this higher-pitched drone into paulstretch, stretched to 50x. i played with the bandwidth and noise/tone sliders to get the distinctive crystalline sound, rendering it a few more times. by playing this tone at different speeds using renoise’s basic sample editor, i was able to layer octaves, fading different copies of the sample in and out for some evolving harmonics as desired.

a signal follower attached to the low-passed kick drum flexed the drone’s volume on the beat, adding some liveliness, resulting in a pleasant low-key “bloom pads” effect. i don’t go for huge sidechain compression; just a touch is all that’s needed to reinforce the rhythm. a slow LFO set to “random” mode, attached to a bitcrusher, downgraded the clap sounds with some pleasant crunch.

calf reverb and vintage tape delay plugins rounded out the FX, with the percussion patterns treated liberally, resulting in some complex sounds despite simple arrangement. the only other effect was a tape warmth plugin on the master channel; everything was kept quite minimal, for aesthetic and time reasons. given that i only had a day or so to work on the track, i knew i couldn’t try for too many complicated tricks or melodies.

Okay this is not going to be a very fun post to read, and the title can already make you think that I’m being an arrogant bastard this time around, but I got a feeling that lately people are missing the point that even being grumpy, I’m not usually grumpy just because, I’m usually grumpy because I’m trying to get things to improve rather than stagnate or get worse.

So let’s take an example right now. Thomáš postd about some of the changes that are to be expected on LibreOffice 4 — one of these is that the LDAP client libraries are no longer an optional dependency but have to be present. I wasn’t happy about that.

I actually stumbled across that just the other day when installing the new laptop: while installing KDE component with the default USE flags, OpenLDAP would have been installed. The reason is obviously that the ldap USE flag is enabled by default, which makes sense, as it’s (unfortunately) the most common “shared address book” database available. But why should I get an LDAPserver if I selected explicitly a desktop profile?

So the first task at hand, was to make sure that the minimal USE flag was present on the package (it was), and if it did what was intended, i.e., not install the LDAP server — and that is the case indeed. Good, so we can install only the client libraries. Unfortunately the default dependencies were slightly wrong, with said USE flag, as some things like libtool (for libltdl) are only really used by the server components. This was easy to fix, together with a couple more fixes.

But as I proposed on the mailing list to change the defaults, for the desktop profile, to have the minimal USE flag enabled, hell broke loose ­— now the good point about it is that the minimal USE flag is definitely being over-used — and I’m afraid I’m at fault there as well, since both NRPE and NSCA have a minimal USE flag. I guess it’s time to reel back on that for me as well. And I now I have a patch to get openldap to gain a server USE flag, enabled by default – except, hopefully, on the desktop profile – to replace the old minimal flag. Incidentally looking into it I also found that said USE flag was actually clashing with the cxx one, for no good reason as far as I could tell. But Robin doesn’t even like the idea of going with a server USE flag for OpenLDAP!

On a different note, let’s take hwids — I originally created the package to reduce the amount of code our units’ firmware required, but while at it I ended up with a problematic file on my hands, as I wrote the oui.txt file downloaded from IEEE has been redistributed for a number of years, but when I contacted them to make sure I could redistribute it, they told me that it wasn’t possible. Unfortunately the new versions of systemd/udev use that file to generate some hardware database — finally implementing my suggestion from four years ago better late than never!

Well, I ended up having to take some flak, and some risk, and now the new hwids package fetches that file (as well as the iab.txt file) and also fully implements re-building the hardware database, so that we can keep it up to date from Portage, without having to get people to re-build their udev package over and over.

So, excuse me if I’m quite hard to work with sometimes, but the amount of crap I have to take when doing my best to make Gentoo better, for users and developers, is so high that sometimes I’d just like to say “screw it” and leave it to someone else to fix the mess. But I’m not doing that — if you don’t see me around much in the next few days, it’s because I’m leaving LA on Wednesday, and I can’t post on the blog while flying to New York (because the gogonet IP addresses are in virtually every possible blacklist, now and in the future - so no way I can post to the blog, unless I figure out a way to set up a VPN and route traffic to my blog to said VPN …).

And believe it or not, but I do have other concerns in my life beside Gentoo.

I’ve promised some insight into how much running the tinderbox actually costed me. And since today marks two months from Google AdSense’s crazy blacklisting of my website, I guess it’s a good a time as any other.

SO let’s start with the obvious first expense: the hardware itself. My original Tinderbox was running on the box I called Yamato, which costed me some €1700 and change, without the harddrives, this was back in 2008 — and about half the cost was paid with donation from users. Over time, Yamato had to have its disks replaced a couple of times (and sometimes the cost came out of donations). That computer has been used for other purposes, including as my primary desktop for a long time, so I can’t really complain about the parts that I had to pay myself. Other devices, and connectivity, and all those things, ended up being shared between my tinderbox efforts and my freelancing job, so I also don’t complain about those in the least.

The new Tinderbox host is Excelsior, which has been bought with the Pledgie which got me paying only some $1200 of my pocket, the rest coming in from the contributors. The space, power and bandwidth, have been offered by my employer which solved quite a few problems. Since now I don’t have t pay for the power, and last time I went back to Italy (in June) I turned off, and got rid of, most of my hardware (the router was already having some trouble; Yamato’s motherboard was having trouble anyway, I saved the harddrive to decide what to do, and sold the NAS to a friend of mine), I can assess how much I was spending on the power bill for that answer.

My usual power bill was somewhere around €270 — which obviously includes all the usual house power consumption as well as my hardware and, due to the way the power is billed in Italy, an advance on the next bill. The bill for the months between July and September, the first one where I was fully out of my house, was for -€67 and no, it’s not a typo, it was a negative bill! Calculator at hand, he actual difference between between the previous bills and the new is around €50 month — assuming that only a third of that was about the tinderbox hardware, that makes it around €17 per month spent on the power bill. It’s not much but it adds up. Connectivity — that’s hard to assess, so I’d rather not even go there.

With the current setup, there is of course one expense that wasn’t there before: AWS. The logs that the tinderbox generates are stored on S3, since they need to be accessible, and they are lots. And one of the reasons why Mike is behaving like a child about me just linking the build logs instead of attaching them, is that he expects me to delete them because they are too expensive to keep indefinitely. So, how much does the S3 storage cost me? Right now, it costs me a whopping $0.90 a month. Yes you got it right, it costs me less than one dollar a month for all the storage. I guess the reason is because they are not stored for high reliability or high speed access, and they are highly compressible (even though they are not compressed by default).

You can probably guess at this point that I’m not going to clear out the logs from AWS for a very long time at this point. Although I would like for some logs not to be so big for nothing — like the sdlmame one that used to use the -v switch to GCC which causes all the calls to print a long bunch of internal data that is rarely useful on a default log output.

Luckily for me (and for the users relying on the tinderbox output!) those expenses are well covered with the Flattr revenue from my blog’s posts — and thank to Socialvest I no longer have to have doubts on whether I should keep the money or use it to flattr others — I currently have over €100 ready for the next six/seven months worth of flattrs! Before this, between my freelancer’s jobs, Flattr, and the ads on the blog, I would also be able to cover at least the cost of the server (and barely the cost of the domains — but that’s partly my fault for having.. a number).

Unfortunately, as I said at the top of the post, there no longer are ads served by Google on my blog. Why? Well, a month and a half ago I received a complain from Google, saying that one post of mine in which I namechecked a famous adult website, in the context of a (at the time) recent perceived security issue, is adult material, and that it goes against the AdSense policies to have ads served on a website with adult content. I would still argue that just namechecking a website shouldn’t be considered adult content, but while I did submit an appeal to Google, a month and a half later I have no response at hand. They didn’t blacklist the whole domain though, they only blacklisted my blog, so the ads are still showed on Autotools Mythbuster (which I count to resume working almost full time pretty soon) but the result is bleak: I went down from €12-€16 a month to a low €2 a month due to this, and that is no longer able to cover for the serve expense by itself.

This does not mean that anything will change in the future, immediate or not. This blog for me has more value than the money that I can get back from it, as it’s a way for me to showcase my ability and, to a point, get employment — but you can understand that it still upsets me a liiiittle bit the way they handled that particular issue.

During the following week there will be hard feature freeze on libreoffice and 4.0 branch will be created. This means that we can finally start to do some sensible stuff, like testing it like hell in Gentoo.

This release is packed with new features so let me list at least some relevant to our Gentoo stuff:

repaired nsplugin interface (who the hell uses it :P) that was fixed by Stephan Bergmann for wich you ALL guys should sent him some cookies :-)

liblangtag direct po/mo usage that ensures easier translations usage because the translations are not converted in to internal sdf format

liborcus library debut which brings out some features from calc to nice small lib so anyone can reuse them, plus it is easier to maintain, cookies to Kohei Yoshida

bluetooth remote control that allows you to just mess with your presentations over bluetooth, also there is android remote app for that over network ;-)

telepathy colaboration framework inclusion that allows you to mess with mutiple other people on one document in semi-realtime manner (it is mostly tech preview and you don’t see what is the other guy doing, it just appears in the doc)

binfilter is gone! Which is awesome as it was huge load of code that was really stinky

For more changes you can just read the wiki article, just keep in mind that this wiki page will be updated until the release, so it does not contain all the stuff.

Build related stuff

We are going to require new library that allows us to parse mspub format. Fridrich Strba was obviously bored so he wrote yet another format parser :-)

Pdfimport is no longer pseudo-extension but it is directly built in with normal useflag, which saves quite a lot of copy&paste code and it looks like it operates faster now.

The openldap schema provider is now hard-required so you can use adresbooks (Mork driver handles that). I bet some of you lads wont like this much, but ldap itself does not have too much deps and it is usefull for quite few business cases.

There are also some nice removals, like glib and librsvg are goners from default reqs (no-suprise for gnomers that they will still need them). From other it no longer needs the sys-libs/db, which I finally removed from my system.

Gcc requirement was raised to 4.6, because otherwise boost acts like *censored* and I have better stuff to do than just fix it all the time.

Saxon buindling has been delt with and removed completely.

Paralel build is sorted out so it will use correct amount of cpus and will fork gcc only required times not n^n times.

And last but most probably worst, the plugin foundation that was in java is slowly migrating to python, and it needs python:3.3 or later. This did not make even me happy :-)

Other fancy libreoffice stuff

Michel Meeks is running merges against the Apache Openoffice so we try hard to get even fixes that are not in our codebase (thankfully allowed by license this way). So with lots of efforts we review all their code changes and try to merge it over into our implementation. This will grow more and more complex over a time, because in libo we actually try to use the new stuff like new C++ std/Boost/… so there are more and more collisions. Lets see how long it will be worth it (of course oneliners are easy to pick up :P).

What is going in stable?

We at last got libreoffice-3.6 and binary stable. After this there was found svg bug with librsvg (see above, its gone from 4.0) so the binaries will be rebuilt and next version bump will loose the svg useflag. This was caused by how I wrote the detection of new switches and overlook on my side, I simply tried to just launch the libreo with -svg and didn’t dig further. Other than that the whole package is production ready and there should not be much new regressions.

So after having to get one laptop to boot on UEFI I got to make the Latitude work with that as well, if anything, as a point of pride. Last time I was able to get Windows booting as UEFI but I ended up using GRUB 2 with the legacy BIOS booting instead of UEFI for Linux, and promisd myself to find a way to set this up before.

Well, after the comments on my previous post I made sure to update my copy of SysRescueCD, as I only had a version 2.x on my USB key the other day, and that does not support EFI booting — but the new version (3.0) actually supports it out of the box, which makes it much easier, as I no longer need the EFI shell to boot an EFI stub kernel. To be precise, there also no need to use EFI stub, if not to help in recovery situations.

So, after booting into SysRescueCD, I zeroed out the Master Boot Record (to remove the old-style GRUB setup), re-typed the first partition to EF00 — it was set to EF02 which is what GRUB2 uses to install its modules on non-EFI systems), and formatted it to vfat, then… I chrooted into the second partition (which is my Gentoo Linux’s root partition), rebuilt GRUB2 to support efi-64, and just used grub2-install. Done!

Yes, the new SysRescueCD makes it absolutely piece of cake. And now I actually disabled the non-UEFI booting of that laptop and, not sure if it’s just my impression, though, it feels like it’s actually a second or two faster.

Still on the UEFI topic, turns out that Fabio ordered the same laptop I got (and I’m writing from right now), which means that soon Sabayon will have to support UEFI booting. On the other hand, I got Gentoo working fine on this laptop and the battery life is great, s I’m not complaining about it too much. I’ll actually write something about the laptop and how it feels soon, but tonight, I’m just too tired for it.

“Nothing gets people’s interest peaked like colorful graphics. Therefore, graphing the web of trust in your local area as you build it can help motivate people to participate as well as giving everyone a clear sense of what’s being accomplished as things progress.”

Last Friday (Black Friday, since I was in the US this year), I ended up buying for myself an early birthday present; I finally got the ZenBook UX31A that I was looking at since September, after seeing the older model being used by J-B of VLC fame. Today it arrived, and I decided to go the easy route: I already prepared a DVD with Sabayon, and after updating the “BIOS” from Windows (since you never know), I wiped it out and installed the new OS on it. Which couldn’t be booted.

Now before you run around screaming “conspiracy”, I ask you to watch Jo’s video (Jo did you really not have anything with a better capture? My old Nikon P50 had a better iris!) and notice that Secure Boot over there works just fine. Other than that, this “ultrabook” is not using SecureBoot because it’s not certified for Windows 8 anyway.

The problem is not that it requires Secure Boot or anything like that but much more simply, it has no legacy boot. Which is what I’m using on the other laptop (the Latitude E6510), since my first attempt at using EFI for booting failed badly. Anyway this simply meant that I had to figure out how to get this to boot.

What I knew from the previous attempt is this:

grub2 supports UEFI both 32- and 64-bit mode, which is good — both my systems run 64-bit EFI anyway;

grub2 requires efibootmgr to set up the boot environment;

efibootmg requires to have access to the EFI variables, so it requires a kernel with support for EFI variables;

but there is no way to access those variables when using legacy boot.

This chicken-and-egg problem is what blown it for me last time — I did try before the kernel added EFI Stub support anyway. So what did I do this time? Well, since Sabayon did not work out of the box I decided to scratch it and I went with good old fashioned Gentoo. And as usual to install it, I started from SysRescueCD — which to this day, as far as I can tell, still does not support booting as EFI either. It’s a good thing then that Asus actually supports legacy boot… from USB drives as well as CDs.

So I boot from SysRescueCD and partition the SSD in three parts: a 200MB, vfat EFI partition; a root-and-everything partition; and a /home partition. Note that I don’t split either /boot or /usr so I’m usually quite easy to please, in the boot process. The EFI partition I mount as /mnt/gentoo/boot/efi and inside it I create a EFI directory (it’s actually case-insensitive but I prefer keeping it uppercase anyway).

Now it’s time to configure and build the kernel — make sure to enable the EFI Stub support. Pre-configure the boot parameters in the kernel, make sure to not use any module for stuff you need during boot. This way you don’t have to care about an initrd at all. Build and install the kernel. Then copy /boot/vmlinuz-* as /boot/efi/EFI/kernel.efi — make sure to give it a .efi suffix otherwise it won’t work — the name you use now is not really important as you’ll only be using it once.

Now you need an EFI shell. The Zenbook requires the shell available somewhere, but I know that at least the device we use at work has some basic support for an internal shell in its firmware. The other Gentoo wiki has a link on where to download the file; just put it in the root of the SysRescueCD USB stick. Then you can select to boot it from the “BIOS” configuration screen, which is what you’ll get after a reboot.

At this point, you just need to execute the kernel stub: FS1:\EFI\kernel.efi will be enough for it to start. After the boot completed, you’re now in an EFI-capable kernel, booted in EFI mode. And the only thing that you’re left to do is grub-install --efi-directory=/boot/grub/efi. And .. you’re done!

When you reboot, grub2 will start in EFI mode, boot your kernel, and be done with it. Pretty painless, isn’t it?

there are a lot of packages assigned to maintainer-needed. This packages lack an active maintainer and his bugs are solved usually by people in maintainer-needed alias (like pinkbyte, hasufell, kensington and me). Even if we are still able to keep the bug list "short" (when excluding "enhancement" and "qa" tagged bugs) any help on this task is really appreciated and, then:1. If you are already a Gentoo Dev and would like to help us, simply join the team adding you to mail alias. There is no need to go to bug list and fix any specified amount of bugs by obligation. For example, I simply try to go to fix maintainer-needed bugs when I have a bit of time after taking care of other things.2. If you are a user, you can:- Step up as maintainer using proxy-maintainers project:http://www.gentoo.org/proj/en/qa/proxy-maintainers/index.xml- Go to bugs:http://tinyurl.com/cssc95vand provide fixes, patches... for them ;)

Today it bit me. I rebooted my workstation, and all hell broke loose. Well, actually, it froze. Literally, if you consider my root file system. When the system tried to remount the root file system read-write, it gave me this:

mount: / not mounted or bad option

So I did the first thing that always helps me, and that is to disable the initramfs booting and boot straight from the kernel. Now for those wondering why I boot with an initramfs while it still works directly with a kernel: it’s a safety measure. Ever since there are talks, rumours, fear, uncertainty and doubt about supporting a separate /usr file system I started supporting an initramfs on my system in case an update really breaks the regular boot cycle. Same because I use lvm on most file systems, and software RAID on all of them. If I wouldn’t have an initramfs laying around, I would be screwed the moment userspace decides not to support this straight from a kernel boot. Luckily, this isn’t the case (yet) so I could continue working without an initramfs. But I digress. Back to the situation.

Booting without initramfs worked without errors of any kind. Next thing is to investigate why it fails. I reboot back with the initramfs, get my read-only root file system and start looking around. In my dmesg output, I notice the following:

EXT4-fs (md3): Cannot change data mode on remount

So that’s weird, not? What is this data mode? Well, the data mode tells the file system (ext4 for me) how to handle writing data to disk. As you are all aware, ext4 is a journaled file system, meaning it writes changes into a journal before applying, allowing changes to be replayed when the system suddenly crashes. By default, ext4 uses ordered mode, writing the metadata (information about files and such, like inode information, timestamps, block maps, extended attributes, … but not the data itself) to the journal right after writing data to the disk, after which the metadata is then written to disk as well.

On my system though, I use data=journal so data too is written to the journal first. This gives a higher degree of protection in case of a system crash (or immediate powerdown – my laptop doesn’t recognize batteries anymore and with a daughter playing around, I’ve had my share of sudden powerdowns). I do boot with the rootflags=data=journal and I have data=journal in my fstab.

But the above error tells me otherwise. It tells me that the mode is not what I want it to be. So after fiddling a bit with the options and (of course) using Google to find more information, I found out that my initramfs doesn’t check the rootflags parameter, so it mounts the root file system with the standard (ordered) mode. Trying to remount it later will fail, as my fstab contains the data=journal tag, and running mount -o remount,rw,data=ordered for fun doesn’t give many smiles.

The man page for genkernel however showed me that it uses real_rootflags. So I reboot with that parameter set to real_rootflags=data=journal and all is okay again.

Edit: I wrote that even changing the default mount options in the file system itself (using tune2fs /dev/md3 -o journal_data) didn’t help. However, that seems to be an error on my part, I didn’t reboot after toggling this, which is apparently required. Thanks to Xake for pointing that out.

Today I have unmasked ghc-7.6.1 in gentoo‘s haskell overlay. Quite a few of things is broken (like unbumped yet gtk2hs), but major things (like darcs) seem to work fine. Feel free to drop a line on #gentoo-haskell to get the thing fixed.

Some notes and events in the overlay:

ghc-7.6.1 is available for all major arches we try to support

a few ebuilds of overlay were converted to EAPI=5 to use subslot depends (see below)

After one month I think it was time to write my review about Gentoo miniconf.

In 20 and 21 October I attended to the Gentoo Miniconf which was a part of the bootstrapping-awesome project, 4 conferences (openSUSE Conference/Gentoo Miniconf/LinuxDays/SUSE Labs) where took place in the Technical Czech University at Prague.

Photo by Martin Stehno

Day 0: After our flight arrived in Prague’s airport – we went straight to the pre-conference welcome party in a cafe near the university where the conference took place. There we met the other greeks who arrived in the previous days and I had also the chance to meet a lot of Gentoo developers and talk with them.

Day 1: The first day started earlier in the morning. Me and Dimitris went to the venue before the conference started in order to prepare the room for the miniconf. The day started with Theo as host to welcome us. There were plenty of interesting presentations that covered a lot of aspects of Gentoo, the Trustees/Council, Public Relations, The Gentoo KDE team, Gentoo Prefix, Security, Catalyst and Benchmarking. The highlight of the day was when Robin Johnson introduced the Infrastructure team and started a very interesting BoF which talked about the state of the Infra team, currently running web apps and the burning issue of the git migration. The first day ended with lots of beers in the big party of the conference in the center of the Prague next to the famous Charles Bridge.

Gentoo Developers group photoPhoto by Jorge Manuel B. S. Vicetto

Day 2:The second day was more relaxed. There were presentations about Gentoo@ IsoHunt, 3D and Linux graphics and Οpen/GnuPG .After the lunch break a Οpen/GnuPG key signing party began outside of the miniconf’s room.After the key signing party we continued with a workshop regarding Puppet also a presentation about how to use testing on Gentoo to improve QA and finally the last presentation ended with Markos and Tomáš talking about how to get involved into development of Gentoo. In the end Theo and Michal closed the session of the miniconf.

I really liked Prague especially the beers and the Czech cuisine.

Gentoo Miniconf was a great exp erience for me. I could write lot of pages about the miniconf because I was in the room the whole days and I saw all the presentations.

I had also the opportunity to get in touch and talk with lots of Gentoo developers and contributors from other FOSS projects. Thanks to Theo and Michal for organizing this awesome event.

More about the presentations and the videos of the miniconf can be found here.Looking forward to the next Gentoo miniconf(why not a conference).

Create a homepage and documentation for a project is a boring task. I
have a few projects that were not released yet due to lack of time and
motivation to create a simple webpage and write down some Sphinx-based
documentation.

To fix this issue I did a quick hack based on my favorite pieces of
software: Flask, docutils and Mercurial. It is a single file web
application that creates homepages automatically for my projects, using
data gathered from my Mercurial repositories. It uses the tags, the
README file, and a few variables declared on the repository's .hgrc
file to build an interesting homepage for each project. I just need to
improve my READMEs! :)

It works similarly to the PyPI Package Index, but accepts any project
hosted on a Mercurial repository, including my non-Python and Gentoo-only
projects.

Last Thursday we had GPG Key & CAcert Signing party at SUSE office inviting anybody who wants to get his key signed. I would say, that it went quite well, we had about 20 people showing up, we had some fun, and we now trust each other some more!

GPG Key Signing

We started with GPG key signing. You know, the ussual stuff. Two rows moving against each other, people exchanging paper slips…

Signing keys

For actually signing keys at home, we recommended people to use signing-party package and caff in particular. It’s easy to use tool as long as you can send mails from command line (there are some options to set up against SMTP directly, but I run into some issues). All you need to do is to call

caff HASH

and it will download the key, show you identities and fingerprint, sign it for you and send each signed identity to the owner by itself vie e-mail. And all that with nice wizard. It can’t be simpler than that.

Importing signatures

When my signed keys started coming back, I was wondering how do I process them. It was simply too many emails. I searched a little bit, but I get too lazy quite soon, so as I have all my mails stored locally in Maildir by offlineimap, I just wrote a following one liner to import them all.

Maybe somebody will find it useful as well, maybe somebody more experienced will tell me in comments how to do it correctly

CAcert

One friend of mine – Theo – really wanted to be able to issue CAcert certificates, so we added CAcert assurance to the program. For those who doesn’t know, CAcert is nonprofit certification authority based on web of trust. You’ll get verified by volunteers and when enough of them trusts you enough, you are trusted by authority itself. When people are verifying you, they give you some points based on how they are trusted and how do they trust you. Once you get 50 points, you are trusted enough to get your certificate signed and once you have 100, you are trusted enough to start verifying other people (after a little quiz to make sure you know what are you doing).

I knew that my colleague Michal čihař is able and willing to issue some points but as he was starting with issuing 10 and I with 15, I also asked few nearby living assurers from CAcert website. Unfortunately I got no reply, but we were organizing everything quite quickly. But we had another colleague – Martin Vidner – showing up and being able to issue some points. I assured another 11 people on the party and now I can give out 25 points. As well as Michal and I guess Martin is now somewhere around 20 as well. So it means that if you need to be able to issue CAcert certificates, visiting just SUSE office in Prague is enough! But still, contact us beforehand, sometimes we do have a vacation

If you’re using LXC, you might have noticed that there was a 0.8.0 release lately, finally, after two release candidates, one of which was never really released. Do you expect everything would go well with it? Hah!

Well, you might remember that over time I found that the way that you’re supposed to mount directories in the configuration files changed, from using the path to use as root, to the default root path used by LXC, and every time that happened, no error message was issued that you were trying to mount directories outside of the tree that they are running.

Last time the problem I hit was that if you try to mount a volume instead of a path, LXC expected you to use as a base path the full realpath, which in the case of LVM volumes is quite hard to know by heart. Yes you can look it up, but it’s a bit of a bother. So I ended up patching LXC to allow using the old format, based on the path LXC mounts it to (/usr/lib/lxc/rootfs). With the new release, this changed again, and the path you’re supposed to use is /var/lib/lxc/${container}/rootfs — again, it’s a change in a micro bump (rc2 to final) which is not documented…. sigh. This would be enough to irk me, but there is more.

The new version also seem to have a bad interaction with the kernel on stopping of a container — the virtual ethernet device (veth pair) is not cleared up properly, and it causes the process to stall, with something insisting to call the kernel and failing. The result is a not happy Diego.

Without even having to add the fact that the interactions between LXC and SystemD are not clear yet – with the maintainers of the two projects trying to sort out the differences between them, at least I don’t have to care about it anytime soon – this should be enough to make it explicit that LXC is not ready for prime time so please don’t ask.

On a different, interesting note, the vulnerability publicized today that can bypass KERNEXEC? Well, unless you disable the net_admin capability in your containers (which also mean you can’t set the network parameters, or use iptables), a root user within a container could leverage that particular vulnerability.. which is one extremely good reason not to have untrusted users having root on your containers.

Oh well, time to wait for the next release and see if they can fix a few more issues.

One of the issues that came through with the recent drama about the n-th udev fork is the matter of assigning copyright to the Gentoo Foundation. This topic is not often explored, mostly because it really is a minefield, and – be ready to be surprised – I think the last person who actually said something sane on the topic has been Ciaran.

Let’s see a moment what’s going on: all ebuilds and eclasses in the main tree, and in most of the overlays, report “Gentoo Foundation” as the holder of copyright. This is so much a requirement that we’re not committing to the tree anything that reports anyone else’s copyright, and we refuse the contribution in that case for the most part. While it’s cargo-culted at this point, it is also an extremely irresponsible thing to do.

First of all, nobody ever signed a copyright assignment form to the Gentoo Foundation, as far as I can tell. I certainly didn’t do it. And especially as we go along with getting more and more proxied maintainers, as they almost always are not Gentoo Foundation members (Foundation membership comes after an year as a developer, if I’m not mistaken — or something along those lines, I honestly forgot because, honestly, I’m not following the Foundation doing at all).

Edit: Robin made me notice that a number of people did sign a copyright assignment, first to Gentoo Technologies that were then re-assigned to the Foundation. I didn’t know that — I would be surprised if a majority of the currently active developers knew about that either. As far as I can tell, copyright assignment was no longer part of the standard recruitment procedure when I joined, as, as I said, I didn’t sign one. Even assuming I was the first guy who didn’t sign it, 44% of the total active developers wouldn’t have signed it, and that’s 78% of the currently active developers (give or take). Make up your mind on these numbers.

But even if we all signed said copyright assignment, it’s for a vast part invalid. The problem with copyright assignment is that they are just that, copyright assignments… which means they only work where the law regime concerning authors’ work is that of copyright. For most (all?) of Europe, the regime is actually that of author’s rights and like VideoLAN shows it’s a bit more complex, as the authors have no real way to “assign” those rights.

Edit²: Robin also pointed at the fact that FSFe, Google (and I add Sun, at the very lest) have a legal document, usually called Contributor License Agreement (when it’s basically replacing a full blown assignment) or Fiduciary Licence Agreement (the more “free software friendly” version). This solves half the problem only, as the Foundation would still not be owning the copyright, which means that you still have to come up with a different way to identify the contributors, as they still have their rights even though they leave any decision regarding their contributions to the entity they sign the CLA/FLA to.

So the whole thing stinks of half-understood problem.

This is actually gotten more complex recently, because the sci team borrowed an eclass (or the logic for an eclass) from Exherbo — who actually handles the individual’s copyright. This is actually a much more sensible approach, on the legal side, although I find the idea of having to list, let’s say, 20 contributors at the top of every 15-lines ebuild a bit of an overkill.

My proposal would then be to have a COPYRIGHTS.gentoo file in every package directory, where we list the contributors to the ebuild. This way even proxied maintainers, and one-time contributors, get their credit. The ebuild can then refer to “see the file” for the actual authors. A similar problem also applies to files that are added to the package, including, but not limited to, the init scripts, and making the file formatted, instead of freeform, would probably allow crediting those as well.

Now, this is just a sketch of an idea — unlike Fabio, whose design methodology I do understand and respect, I prefer posting as soon as I have something in mind, to see if somebody can easily shoot it down or if it has wings to fly, and also in the vain hope that if I don’t have the time, somebody else would pick up my plan — but if you have comments on it, I’d be happy to hear them. Maybe after a round of comments, and another round of thinking about it, I’ll propose it as a real GLEP.

During the last years I started several open source projects. Some turned out to be useful, maybe successful, many were just rubbish. Nothing new until here.

Every time I start a new project, I usually don’t really know where I am headed and what my long-term goals are. My excitement and motivation tipically come from solving simple everyday and personal problems or just addressing {short,mid}-term goals. This is actually enough for me to just hack hack hack all night long. There is no big picture, no pressure from the outside world, no commitment requirements. It’s just me and my compiler/interpreter having fun together. I call this the “initial grace period”.

During this period, I usually never share my idea with other people, ever. I kind of keep my project in a locked pod, away from hostile eyes. Should I share my idea at this time, the project might get seriously injured and my excitement severely affected. People would only see the outcome of my thought, but not the thought process itself nor detailed plans behind it, because I just don’t have them! Besides this might be both considered against any basic Software Engineering rules or against some exotic “free software” principles, it works for me.

I don’t want my idea to be polluted as long as I don’t have something that resembles it in the form of a consistent codebase. And until that time, I don’t want others to see my work and judge its usefulness basing on incomplete or just inconsistent pieces of information.

At the very same time, writing documents about my idea and its goals beforehand is also a no-go, because I have “no clue” myself as mentioned earlier.

This is why revision control systems and the implicit development model they force on individuals are so important, especially for me.
Giving you the ability to code on your stuff, changes, improvements, without caring about the external world until you are really really done with it, is what I ended up needing so so much.
Every time I forgot to follow this “secrecy” strategy, I had to spend more time discussing about my (still confused?) idea on {why,what,how} I am doing than coding itself. Round trips are always expensive, no matter what you’re talking about!

Many internal tools we at Sabayon successfully use have gone through this development process. Other staffers sometimes tell things like “he’s been quiet in the last few days, he must be working on some new features”, and it turns out that most of the times this is true.

This is what I wanted to share with you today though. Don’t wait for your idea to become clearer in your mind, it won’t happen by itself. Just take a piece of paper (or your text editor), start writing your own secret goals (don’t make the mistake of calling them “functional requirements” like I did sometimes), divide them into modest/expected and optimistic/crazy and start coding as soon as possible on your own version/branch of the repo. Then go back to your list of goals, see if they need to be tweaked and go back coding again. Iterate until you’re satisfied of the result, and then, eventually, let your code fly away to some public site.

But, until then, don’t tell anybody what you’re doing! Don’t expect any constructive feedback during the “initial grace period”, it is very likely that it will be just be destructive.

I spent half my Saturday afternoon working on Blender, to get the new version (2.64a) in Portage. This is never an easy task but in this case it was a bit more tedious because thanks to the new release of libav (version 9) I had to make a few more modifications … making sure it would still work with the old libav (0.8).

FFmpeg support is not guaranteed — If you care you can submit a patch. I’m one of the libav developers thus that’s what I work, and test, with.

Funnily enough, while I was doing that work, a new bug for blender was reported in Gentoo, so I looked into it and found out that it was actually caused by one of the bundled dependencies — luckily, one that was already available as its own ebuild, so I just decided to get rid of it. The interesting part was that it wasn’t listed in the “still bundled libraries” list that the ebuild’s own diagnostic prints… since it was actually a bundled library of the bundled libmv!

So you reach the point where you get one package (Blender) bundling a library (libmv) bundling a bunch of libraries, multi-level.

Looking into it I found out that not only the dependency that was causing the bug was bundled (ldl) but there were at least two more that, I knew for sure, were available in Gentoo (glog and gflags). Which meant I could shave some more code out of the package, by adding a few more dependencies… which is always a good thing in my book (and I know that my book is not the same as many others’).

While looking for other libraries to unbundle, I found another one, mostly because its name (eltopo) was funny — it has a website and from there you can find the sources — neither are linked in the Blender package. When I looked at the sources, I was dismayed to see that there was no real build system but just an half-broken Makefile building two completely different PIC-enabled static archives, for debug and release. Not really something that distributions could get much interest in packaging.

So I set up at build my usual autotools-based build system (which no matter what people say it’s extremely fast, if you know how to do it), fix the package to build with gcc 4.7 correctly (how did it work for Blender? I assume they patched it somehow but they don’t write down what they do!), and .. uh where’s the license file?

Turns out that while the homepage says that the library is “public domain”, there is no license statement anywhere in the source code, making it in all effects the exact opposite: a proprietary software. I’ve opened an issue for it and hopefully upstream will fix that one up so I can send him my fixes and package it in Gentoo.

Interestingly enough, the libmv software that Blender packages, is much better in its way of bundling libraries. While they don’t seem to give you an easy way to disable the bundled copies (which might or might not be Blender’s build system fault), they make it clear where each library come from, and they have scripts to “re-bundle” said libraries. When they make changes, they also keep a log of them so that you can identify what changed and either ignore, patch or send it upstream. If all projects bundling stuff did it that way, it would be a much easier job to unbundle…

In the mean time, if you have some free time and feel like doing something to improve the bundled libraries situation in Gentoo Linux, or you care about Blender and you’d like to have a better Gentoo experience with it, we could use some ebuilds for ceres-solver and SSBA as well as fast-C (this last one has no buildsystem at all, unfortunately) all used by libmv, or maybe carve libredcode (for which I don’t even have an URL at hand), recastnavigation (which has no releases) which are instead used directly by Blender.

P.S.: don’t expect to see me around this Sunday, I’m actually going to see the Shuttle, and so I won’t be back till late, most likely, or at least I hope so. You’ll probably see a photo set on Monday on my Flickr page if you want to have a treat.

This wednesday, the Gentoo Hardened team held its monthly online meeting, discussing the things that have been done the last few weeks and the ideas that are being worked out for the next. As I did with the last few meetings, allow me to summarize it for all interested parties…

Toolchain

The upstream GCC development on the 4.8 version progressed into its 3rd stage of its development cycle. Sadly, many of our hardened patches didn’t make the release. Zorry will continue working on these things, hopefully still being able to merge a few – and otherwise it’ll be for the next release.

For the MIPS platform, we might not be able to support the hardenedno* GCC profiles [1] in time. However, this is not seen as a blocker (we’re mostly interested in the hardened ones, not the ones without hardening ;-) so this could be done later on.

Blueness is migrating the stage building for the uclibc stages towards catalyst, providing more clean stages. For the amd64 and i686 platforms, the uclibc-hardened and uclibc-vanilla stages are already done, and mips32r2/uclibc is on the way. Later, ARM stages will be looked at. Other platforms, like little endian MIPS, are also on the roadmap.

Kernel

The latest hardened-sources (~arch) package contains a patch supporting the user.* namespace for extended attributes in tmpfs, as needed for the XATTR_PAX support [2]. However, this patch has not been properly investigated nor tested, so input is definitely welcome. During the meeting, it was suggested to cap the length of the attribute value and only allow the user.pax attribute, as we are otherwise allowing unprivileged applications to “grow data” in the kernel memory space (the tmpfs).

Prometheanfire confirmed that recent-enough kernels (3.5.4-r1 and later) with nested paging do not exhibit the performance issues reported earlier.

SELinux

The 20120725 upstream policies are stabilized on revision 5. Although a next revision is already available in the hardened-dev overlay, it will not be pushed to the main tree due to a broken admin interface. Revision 7 is slated to be made available later the same day to fix this, and is the next candidate for being pushed to the main tree.

The september-released newer userspace utilities for SELinux are also going to be stabilized in the next few days (at the time of writing this post, they are ;-). These also support epatch_user so that users and developers can easily add in patches to try out stuff without having to repackage the application themselves.

grSecurity and PaX

The toolchain support for PT_PAX (the ELF-header based PaX markings) is due to be removed soon, meaning that the XATTR_PAX support will need to be matured by then. This has a few consequences on available packages (which will need a bump and fix) such as elfix, but also on the pax-utils.eclass file (interested parties are kindly requested to test out the new eclass before it reaches “production”). Of course, it will also mean that the new PaX approach needs to be properly documented for end users and developers.

pipacs also mentioned that he is working on a paxctld daemon. Just like SELinux’ restorecond daemon, this deamon will look for files and check them against a known database of binaries with their appropriate PaX markings. If the markings are set differently (or not set), the paxctld daemon will rectify the situation. For Gentoo, this is less of a concern as we already set the proper information through the ebuilds.

Profiles

The old SELinux profiles, which were already deprecated for a while, have been removed from the portage tree. That means that all SELinux-using profiles use the features/selinux inclusion rather than a fully build (yet difficult to maintain) profile definition.

System Integrity

A few packages, needed to support or work with ima/evm, have been pushed to the hardened-dev overlay.

So again a nice month of (volunteer) work on the security state of Gentoo Hardened. Thanks again to all (developers, contributors and users) for making Gentoo Hardened where it is today. Zorry will send out the meeting log later to the mailinglist, so you can look at the more gory details of the meeting if you want.

[1] GCC profiles are a set of parameters passed on to GCC as a “default” setting. Gentoo hardened uses GCC profiles to support using non-hardening features if the users wants to (through the gcc-config application).

[2] XATTR_PAX is a new way of handling PaX markings on binaries. Previously, we kept the PaX markings (i.e. flags telling the kernel PaX code to allow or deny specific behavior or enable certain memory-related hardening features for a specific application) as flags in the binary itself (inside the ELF header). With XATTR_PAX, this is moved to an extended attribute called “user.pax”.

Few days ago I finished fiddling with open build service (obs) packages in our main tree. Now when anyone wants to mess up with obs he just have to emerge dev-util/osc and have the fun with it.

What the hell is obs?

OBS is pretty cool service that allows you to specify how to build your package and its dependencies in one .spec file where you can deliver the results to multiple archs/distros and not care about how it happens (Debian, SUSE, Fedora, CentOS, Archlinux).

Primary implementation is running for SUSE and it is free to use by anyone (eg. you don’t have to build suse packages there if you don’t want to :P). It has two ways how to interact with the whole tool, one is the web application, which is really PITA and the other is the osc command line tool I finished fiddling with.

Okay so why did you do it?

Well I work at SUSE and we are free to use whatever distro we want while being able to complete our taks. I like to improve stuff I want to be able fix bugs in SLE/openSUSE while not having any chroot/virtual with the named system installed, for such task this works pretty well :-)

The recruiters team announced a few months ago that they decided not to use the recruiting webapp any more, and move back to the txt quizes instead. Additionally, the webapp started showing random ruby exceptions, and since nobody is willing to fix them, we found it a good opportunity to shut down the service completely. There have been people that were still working on it though (including me), so if you are a mentor, mentee or someone who had answers in there, please let me know so I can extract your data and send it to you.
And now I’d like to state my personal thoughts regarding the webapp and the recruiter’s decision to move back to the quizes. First of all, I used this webapp as mentor a lot from the very first point it came up, and I mentored about 15 people through it. It was a really nice idea, but not properly implemented. With the txt quizes, the mentees were sending me the txt files by mail, then we had to schedule an IRC meeting to review the answers, or I had to send the mail back etc. It was a hell for both me and the mentee. I was ending up with hundreds of attachments, trying to find out the most recent one (or the previous one to compare answers), and the mentee had to dig between irc logs and mails to find my feedback.
The webapp solved that issue, since the mentee was putting his answers in a central place, and I could easily leave comments there. But it had a bunch of issues though, mostly UI related. It required too many clicks for simple actions, the notification system was broken by design, I had no easy way to see diffs or to see the progress of my mentee (answers replied / answers left). For example, in order to approve an answer, I had to press “Edit” which transfered me in a new page, where I had to tick “Approve” and press save. Too much, I just wanted to press “Approve”! When I decided to start filling bugs, surprisingly I found out that all my UI complaints had already been reported, clearly I was not alone in this world.
In short, cool idea but annoying UI. That was not the problem though, the real problem is that nobody was willing to fix those issues, which led to the recruiters’ decision to move back to txt quizes. But I am not going back to the txt quizes, no way. Instead, I will start a Google doc and tell my mentees to put their answers there. This will allow me to write my comments below their answers with different font/color, so I can have async communication with them. I was present during the recruitment interview session of my last mentee Pavlos, and his recruiter Markos fired up a Google doc for some coding answers, and it worked pretty well. So I decided to do the same. If the recruiters want the answers in plain text, fine, I can extract them easily.
I’d like to thank a lot Joachim Bartosik, for his work on the webapp and his interesting ideas he put on this (it saved me a lot of time, and made the mentoring process fun again), and Petteri Räty who mentored Joachim creating the recruiting webapp as GSoC project, and helped in deploying it to infra servers. I am kinda sad that I had to shut it down, and I really hope that someone steps up and revives it or creates an alternative. There has been some discussion regarding that webapp during the Gentoo Miniconf, I hope it doesn’t sink.

So we use roughly 1/3rd the memory to get the same things done (fileserver),
and an informal performance analysis gives us roughly double the IO throughput.
On the same hardware!
(The IO difference could be attributed to the ext3 -> ext4 upgrade and the kernel 2.6.18 -> 3.2.1 upgrade)

Another random data point: A really clumsy mediawiki (php+mysql) setup.
Since php is singlethreaded the performance is pretty much CPU-bound;
and as we have a small enough dataset it all fits into RAM.
So we have two processes (mysql+php) that are serially doing things.

And a "move data around" comparison: 63GB in 3.5h vs. 240GB in 4.5h - or roughly 4x the throughput

So, to summarize: For the same workload on the same hardware we're seeing substantial improvements
between a few percent and roughly four times the throughput, for IO-bound as well as for CPU-bound tasks.
The memory use goes down for most workloads while still getting the exact same results, only a lot faster.

App developers and end users both like bundled software, because it’s easy to support and easy for users to get up and running while minimizing breakage. How could we come up with an approach that also allows distributions and package-management frameworks to integrate well and deal with issues like security? I muse upon this over at my RedMonk blog.

Apparently it’s been a while since my last blog post. This however does mean that I’ve been too busy on the coding side, which is what you may prefer I guess.

The new Equo code is hitting the main Sabayon Entropy repository as I write. But what’s it about?

Refactoring

First thing first. The old codebase was ugly, as in, really ugly. Most of it was originally written in 2007 and maintained throughout the years. It wasn’t modular, object oriented, bash-completion friendly, man pages friendly, and most importantly, it did not use any standard argument parsing library (because there was no argparse module and optparse was about to be deprecated).

Modularity

Equo subcommands are just stand-alone modules. This means that adding new functionality to Equo is only a matter of writing a new module, containing a subclass of “SoloCommand” and registering it against the “command dispatcher” singleton object. Also, the internal Equo library has now its own name: Solo.

Backward compatibility

In terms of command line exposed to the user, there are no substantial changes. During the refactoring process I tried not to break the current “equo” syntax. However, syntax that has been deprecated more than 3 years ago is gone (for instance, stuff like: “equo world”). In addition, several commands are now sporting new arguments (have a look at “equo match” for example).

Man pages

All the equo subcommands are provided with a man page which is available through “man equo-<subcommand name>”. The information required to generated the man page is tightly coupled with the module code itself and automatically generated via some (Python + a2x)-fu. As you can understand, maintaining both the code and its documentation becomes easier this way.

Bash completion

Bash completion code lives together with the rest of the business logic. Each subcommand exposes its bash completion options through a class instance method called “list bashcomp(last_argument_str)”, overridden from SoloCommand. In layman’s terms, you’ve got working bashcomp awesomeness for every equo command available.

Where to go from here

Tests, we need more tests (especially regression tests). And I have this crazy idea to place tests directly in the subcommand module code.
Testing! Please install entropy 149 and play with it, try to break it and report bugs!

I'm sitting on the first day of the Qt Developer Days in Berlin and am pretty
impressed about the event so far -- the organizers have done an excellent job
and everything feels very, very smooth here. Congratulations for that; I have a
first-hand experience with organizing a workshop and can imagine the huge pile of
work which these people have invested into making it rock. Well done I say.

It's been some time since I blogged about Trojitá, a fast and lightweight IMAP
e-mail client. A lot of work has found the way in since the last release; Trojitá now supports
almost all of the useful IMAP extensions including QRESYNC and
CONDSTORE for blazingly fast mailbox synchronization or the
CONTEXT=SEARCH for live-updated search results to name just a few.
There've also been roughly 666 tons of bugfixes, optimizations, new features and
tweaks. Trojitá is finally showing evidence of getting ready for being usable as
a regular e-mail client, and it's exciting to see that process after 6+ years of
working on that in my spare time. People are taking part in the development
process; there has been a series of commits from Thomas Lübking of the kwin fame
dealing with tricky QWidget issues, for example -- and it's great to see many
usability glitches getting addressed.

The last nine months were rather hectic for me -- I got my Master's degree
(the thesis was about
Trojitá, of course), I started a new job (this time using Qt) and
implemented quite some interesting stuff with Qt -- if you have always wondered
how to integrate Ragel, a parser generator, with qmake, stay tuned for future
posts.

Anyway, in case you are interested in using an extremely fast e-mail client
implemented in pure Qt, give Trojitá a try. If you'd like to chat about it,
feel free to drop me a mail or just stop me
anywhere. We're always looking for contributors, so if you hit some annoying
behavior, please do chime in and start hacking.

I’m not sure if you’re following the development of this particular package in Gentoo, but with some discussion, quite a few developers reached a consensus last week that the slotted dev-libs/boost that we’ve had for the past couple of years had to go, replaced with a single-slot package like we have for most other libraries.

The main reason for this is that the previous slotting was not really doing what the implementers expected it to do — the idea for many is that you can always depend on whatever highest version of Boost you support, and if you don’t support the latest, no problem, you’ll get an older one. Unfortunately, this clashes with the fact that only the newest version of Boost is supported by upstream with modern configurations, so it happens that a new C library, or a new compiler, can (and do) make older versions non-buildable.

Like what happened with the new GLIBC 2.16, which is partially described in the previous post of the same series, and lately summarized, where there’s no way to rebuild boost-1.49 with the new glibc (the “patch” that could be used would change the API, making it similar to boost-1.50 which ..), but since I did report build failures with 1.50, people “fixed” them by depending on an older version… which is now not installable. D’oh!

So what did I do to sort this out? We dropped the slot altogether. Now all Boost versions install as slot zero and each replace the others. This makes it much easier for both developers and users, as you know that the one version you got installed is the one you’re building against, instead of “whatever has been eselected” or “whatever was installed last” or “whatever is the first one that the upstream user is finding” which was before — usually a mix of all.

But this wasn’t enough because unfortunately, libraries, headers and tools were all slotted so they all had different names based on the version. This was handled in the new 1.52 release which I unmasked today, by going back to the default install layout that Boost uses for Unix: the system layout. This is designed to allow one and only one version of each Boost library in the system, and does neither provide a version nor a variant suffix. This meant we needed another change.

Before going back to system layout, each boost version installed two sets of libraries, one that was multithread-safe and oen that wasn’t. Software using threads would have to link to the mt variant, while those not using threads could link to the (theoretically lower-overhead) single-thread variant. Which happened to be the default. Unfortunately, this also meant that a ton of software out there, even when using threads, simply linked to the boost library they wanted without caring for the variant. Oopsie.

Even worse, it was very well possible, and indeed was the case for Blender, that both variants were brought in, in the process’s address space, possibly causing extremely hard to debug issues due to symbol collisions (which I know, unfortunately, very well).

An easy way to see (using older versions of boost ebuilds) whether your program is linking to the wrong variant, is to see if you see it linking to libboost_threads-mt and at the same time to some other library such as libboost_system (not mt variant). Since our very pleasant former maintainer decided to link the mt variant of libboost_threads to the non-mt one, quite a few ways to check for multithreaded Boost simply … failed.

Now the decision on whether to build threadsafe or not is done through an USE flag like most other ebuilds do, and since only one variant is installed, everybody gets, by default and in most cases, the multithread-safe version, and all is good. Packages requiring threads might want already to start using dev-libs/boost[threads(+)] to make sure that they are not installed with a non-threadsafe version of Boost, but there are symlinks in place righ tnow so that even if they are looking for the mt variant they get the one installed version of boost anyway (only with USE=threads of course).

One question that raised was “how broken will people’s systems be, after upgrading from one Boost to another?” and the answer is “quite” … unless you’re using a modern enough Portage (the last few versions of the 2.1 series are okay, and most of the 2.2), which can use preserve-libs. In that case, it’ll just require you to run a single emerge command to get back on the new version, and if not, you’ll have to wait for revdep-rebuild to finish.

And to make things sweeter, with this change, the time it takes for Boost to build is halved (4 minutes vs 8 on my laptop), while the final package is 30MB less (here at least), since only one set of libraries is installed instead of two — without counting the time and space you’d waste by having to install multiple boost versions together.

And for developers, this also mean that you can forget about the ruddy boost-utils.eclass, since now everything is supposed to work without any trickery. Win-win situation, for once.

So after my descriptive post you might be wondering what’s so complex or time-requiring in running a tinderbox. That’s because I haven’t spoken about the actual manual labor that goes into handling the tinderbox.

The major work is of course scouring the logs to make sure that I file only valid bugs (and often enough that’s not enough, as things hide behind the surface), but there are a quite a number of tasks that are not related to the bug filing, at least not directly.

First of all, there is the matter of making sure that the packages are available for installation. This used to be more complex, but luckily thanks to REQUIRED_USE and USE deps, this task is slightly easier than before. The tinderbox.py script (that generates the list of visible packages that need to be tested) also generates a list of use conflicts, dependencies etc. This list I have to look at manually, and then update the package.use file so that they are satisfied. If their dependencies or REQUIRED_USE are not satisfied, the package is not visible, which means it won’t be tested.

This sounds extremely easy, but there are quite a few situations, which I discussed previously where there is no real way to satisfy requirements for all the packages in the tree. In particular there are situations where you can’t enable the same USE flag all over the tree — for instance if you do enable icu for libxml2, you can’t enable it for qt-webkit (well, you can but you have to disable gstreamer then, which is required by other packages). Handling all the conflicting requirements takes a bit of trial and error.

Then there is a much worse problem and that is with tests that can get stuck, so that things like this happen:

And I’ve got to keep growing the list of packages whose tests are unreliable — I wonder if the maintainers ever try running their tests, sometimes.

This task used to be easier because the tinderbox supports sending out tweets or dents through bti so that it would tell me what was its action — unfortunately identi.ca kept marking the tinderbox’s account as spam, and while they did unlock it three times, it meant I had to ask support to do so every other week. I grew tired of that and stopped caring about it. Unfortunately that means I have to connect to the instance(s) from time to time to make sure they are still crunching.

i received my native instruments komplete audio 6 in the mail today. i wasted no time plugging it in. i have a few first impressions:

build quality

this thing is heavy. not unduly so — just two or three times heaver than the audiofire 2 it replaces. it’s solidly built, so i imagine it can take a fair amount of beating on-the-go. knobs are sturdy, stiff rather than loose, without much wiggle. the big top volume knob is a little looser, with more wiggle, but it’s also made out of metal, rather than the tough plastic of the front trim knobs. the input ports grip 1/4″ jacks pretty tightly, so there’s no worry that cables will fall out.

i haven’t tested the main outputs yet, but the headphone output works correctly, offering more volume than my ears can take, and it seems to be very quiet — i couldn’t hear any background hiss even when turning up the gain.

JACK support

i have mixed first impressions here. according to ALSA upstream, and one of my buddies who’s done some kernel driver code for NI interfaces, it should work perfectly, as it’s class-compliant to the USB2.0 spec (no, really, there is a spec for 2.0, and the KA6 complies with it, separating it from the vast majority of interfaces that only comply with the common 1.1 spec).

i setup some slightly more aggressive settings on this USB interface than for my FireWire audiofire 2, which seems to have been discontinued in favor of echo’s new USB interface (though the audiofire 4 is still available, and is mostly the same). i went with 64 frames/period, 48000 sample rate, 3 periods/buffer . . . which got me 4ms latency. that’s just under half the 8ms+ latency i had with the firewire-based af2.

at these settings, qjackctl reported about 18-20% CPU usage, idling around 0.39-5.0% with no activity. i only have a 1.5ghz core2duo processor from 2007, so any time the CPU clocks down to 1.0ghz, i expect the utilization numbers to jump up. switching from the ondemand to performance governor helps a bit, raising the processor speed all the way up.

playing a raw .wav file through mplayer’s JACK output worked just fine. next, i started ardour 3, and that’s where the troubles began. ardour has shown a distressing tendency to crash jackd and/or the interface, sometimes without any explanation in the logs. one second the ardour window is there, the next it’s gone.

i tried renoise next, and loaded up an old tracker project, from my creative one-a-day: day 316, beta decay. this piece isn’t too demanding: it’s sample-based, with a few audio channels, a send, and a few FX plugins on each track.

playing this song resulted in 20-32% CPU utilization, though at least renoise crashed less often than ardour. renoise feels noticeably more stable than the snapshot of ardour3 i built on july 9th.

i wasn’t very thrilled with how much work my machine was doing, since the CPU load was noticeably better with the af2. though this is to be expected; the CPU doesn’t have to do so much processing of the audio streams; the work is offloaded onto the firewire bus. with usb, all traffic goes through the CPU, so that takes more valuable DSP resources.

still, time to up the ante. i raised the sample rate to 96000, restarted JACK, and reloaded the renoise project. now i had 2ms latency…much lower than i ever ran with the af2. this low latency took more cycles to run, though: CPU utilization was between 20% and 36%, usually around 30-33%.

i haven’t yet tested the device on my main workstation, since that desktop computer is still dead. i’m planning to rebuild it, moving from an old AMD dualcore CPU to a recent Intel Ivy Bridge chip. that should free up enough resources to create complex projects while simultaneously playing back and recording high-quality audio.

first thoughts

i’m a bit concerned that for a $200 best-in-class USB2.0 class-compliant device, it’s not working as perfectly as i’d hoped. all 6/6 inputs and outputs present themselves correctly in the JACK window, but the KA6 doesn’t show up as a valid ALSA mixer device if i wanted to just listen to music through it, without running JACK.

i’m also concerned that the first few times i plug it in and start it, it’s mostly rock-solid, with no xruns (even at 4ms) appearing unless i run certain (buggy) applications. however, it’s xrun/crash-prone at a sample rate of 96000, forcing me to step down to 48000. i normally work at that latter rate anyway, but still…i should be able to get the higher quality rates. perhaps a few more reboots might fix this.

it could be one of the three USB ports on this laptop shares a bus with another high-traffic device, which means there could be bandwidth and/or IRQ conflicts. i’m also running kernel 3.5.3 (ck-sources), with alsa-lib 1.0.25, and there might have been driver fixes in the 3.6 kernel and alsa-lib 1.0.26. i’m also using JACK1, version 0.121.3, rather than the newer JACK2. after some upgrades, i’ll do some more testing.

early verdict: the KA6 should work perfectly on linux, but higher sample rates and lowest possible latency are still out of reach. sound quality is good, build quality is great. ALSA backend support is weak to nonexistent; i may have to do considerable triage and hacking to get it to work as a regular audio playback device.

If you use the slock application, like I do, you may have noticed a subtle change with the latest release (which is version 1.1). That change is that the background colour is now teal-like when you start typing your password in order to disable slock, and get back to using your system. This change came from a dual-colour patch that was added to version 1.1.

I personally don’t like the change, and would rather have my screen simply stay black until the correct password is entered. Is it a huge deal? No, of course not. However, I think of it as just one additional piece of security via obscurity. In any case, I wanted it back to the way that it was pre-1.1. There are a couple ways to accomplish this goal. The first way is to build the package from source. If your distribution doesn’t come with a packaged version of slock, you can do this easily by downloading the slock-1.1 tarball, unpacking it, and modifying config.mk accordingly. The config.mk file looks like this:

but note that you do not need the extra set of escaping backslashes when you are using the colour name instead of hex representation.

If you use Gentoo, though, and you’re already building each package from source, how can you make this change yet still install the package through the system package manager (Portage)? Well, you could try to edit the file, tar it up, and place the modified tarball in the /usr/portage/distfiles/ directory. However, you will quickly find that issuing another emerge slock will result in that file getting overwritten, and you’re back to where you started. Instead, the package maintainer (Jeroen Roovers), was kind enough to add the ‘savedconfig’ USE flag to slock on 29 October 2012. In order to take advantage of this great USE flag, you firstly need to have Portage build slock with the USE flag enabled by putting it in /etc/portage/package.use:

echo "x11-misc/slock savedconfig" >> /etc/portage/package.use

Then, you are free to edit the saved config.mk which is located at /etc/portage/savedconfig/x11-misc/slock-1.1. After recompiling with the ‘savedconfig’ USE flag, and the modifications of your choice, slock should now exhibit the behaviour that you anticipated.

email me "proof" you are running the latest stable -rc kernel at the moment.

send a link to some kernel patches you have done that were accepted into Linus's tree.

send a link to any Linux distro kernel tree where they keep their patches.

say why you want to do this type of thing, and what amount of time you can spend on it per week.

I'll close the application process in a week, on November 7, 2012, after that
I'll contact everyone who applied and do some follow-up questions through email
with them. I'll also post something here to say what the response was like.

This problem seems to bite some of our hardened users a couple of times a year, so thought I’d blog about it. If you are using grsec and PulseAudio, you must not enable CONFIG_GRKERNSEC_SYSFS_RESTRICT in your kernel, else autodetection of your cards will fail.

PulseAudio’s module-udev-detect needs to access /sys to discover what cards are available on the system, and that kernel option disallows this for anyone but root.

A few days ago the box that was hosting our low-risk webapps died (barbet.gentoo.org). The services that were affected are get.gentoo.orgplanet.gentoo.orgpackages.gentoo.orgdevmanual.gentoo.orginfra-status.gentoo.org and bouncer.gentoo.org. We quickly migrated the services to another box (brambling.gentoo.org). Brambling had issues in the past with its RAM, but we changed them with new ones a couple of months ago. Additionally, this machine was used for testing only. Unfortunately the machine started to malfunction as soon as those services were transferred there, which means that it has more hardware issues than the RAM. The resulting error messages stopped when we disabled packages.gentoo.org temporarily. The truth is that this packages webapp is old, unmaintained, uses deprecated interfaces and real pain to debug. In this year’s GSoC we had a really nice replacement by Slava Bacherikov written in django. Additionally, recently we were given a Ganeti cluster hosted at OSUOSL. Thus we decided not to put up again the old packages.gentoo.org instance, and instead create 4 virtual machines in our Ganeti cluster, and migrate the above webapps there, along with the new and shiny packages.gentoo.org website. Furthermore, we will also deploy another GSoC webapp, gentoostats, and start providing our developers with virtual machines. We will not give public IPv4 IPs to the dev VMs though, but probably use IPv6 only so that developers can access them through woodpecker (the box where the developers have their shell accounts), but it is still under discussion. We already started working on the above, and we expect next week to be fully finished with the new webapps live and rocking. Special thanks to Christian and Alec who took care of the migrations before and during the Gentoo Miniconf.

A couple of days ago, Tomas and I, gave a presentation at the Gentoo Miniconf. The subject of the presentation was to give an overview of the current recruitment process, how are we performing compared to the previous years and what other ways there are for users to help us improve our beloved distribution. In this blog post I am gonna get into some details that I did not have the time to address during the presentation regarding our recruitment process.

Recruitment Statistics from 2008 to 2012

Looking at the previous graph, two things are obvious. First of all, every year the number of people who wanted to become developers is constantly decreased. Second, we have a significant number of people who did not manage to become developers. Let me express my personal thoughts on these two things.

For the first one, my opinion is that these numbers are directly related to the Gentoo’s reputation and its “infiltration” to power users. It is not a secret that Gentoo is not as popular as it used to be. Some people think this is because of the quality of our packages, or because of the frequency we cause headaches to our users. Other people think that the “I want to compile every bit of my linux box” trend belongs to the past and people want to spend less time maintaining/updating their boxes and more time doing some actual work nowadays. Either way, for the past few years we are loosing people, or to state it better, we are not “hiring” as many as we used to. Ignoring those who did not manage to become developers, we must admit that the absolute numbers are not in our favor. One may say that, 16 developers for 2011-2012 is not bad at all, but we aim for the best right? What bothers me the most is not the number of the people we recruit, but that this number is constantly falling for the last 5 years…

As for the second observation, we see that, every year, around 4-5 people give up and decide to not become developers after all. Why is that? The answer is obvious. Our long, painful, exhausting recruitment process drives people away. From my experience, it takes about 2 months from the time your mentor opens your bug, until a recruiter picks you up. This obviously kills someone’s motivation, makes him lose interest, get busy with other stuff and he eventually disappears. We tried to improve this process by creating a webapp two years ago, but it did not work out well. So we are now back to square one. We really can’t afford loosing developers because of our recruitment process. It is embarrassing to say at least.

Again, is there anything that can be done? Definitely yes. I’d say, we need an improved or a brand new web application that will focus on two things:

1) make the review process between mentor <-> recruit easier

2) make the final review process between recruit <-> recruiter an enjoyable learning process

Ideas are always welcomed. Volunteers and practical solutions even more ;) In the meantime, I am considering using Google+ hangouts for the face-to-face interview sessions with the upcoming recruits. This should bring some fresh air to this process ;)

The Gentoo Miniconf is over now but it was a great success. There was 30+ developers that went and I met quite some users too. Thanks to Theo (tampakrap) and Michal (miska) for organizing the event (and others), thanks to openSUSE for sponsoring and letting the Gentoo Linux guys hangout there. Thanks to the other sponsors too, Google, Aeroaccess, et al.

For about a year now, I’ve been working at GRNET on its (OpenStack API compliant) open source IaaS cloud platform Synnefo, which powers the ~okeanos service.

Since ~okeanos is mainly aimed towards the Greek academic community (and thus has restrictions on who can use the service), we set up a ‘playground’ ‘bleeding-edge’ installation (okeanos.io) of Synnefo, where anyone can get a free trial account, experiment with the the Web UI, and have fun scripting with the kamaki API client. So, you get to try the latest features of Synnefo, while we get valuable feedback. Sounds like a fair deal.

Unfortunately, being the only one in our team that actually uses Gentoo Linux, up until recently Gentoo VMs were not available. So, a couple of days ago I decided it was about time to get a serious distro running on ~okeanos (the load of our servers had been ridiculously low after all ). For future reference, and in case anyone wants to upload their own image on okeanos.io or ~okeanos, I’ll briefly describe the steps I followed.

4) Chroot and install Gentoo in /mnt/gentoo. Just follow the handbook. At a minimum you’ll need to extract the base system and portage, and set up some basic configs, like networking. It’s up to you how much you want to customize the image. For the Linux Kernel, I just copied directly the Debian /boot/[vmlinuz|initrd|System.map] and /lib/modules/ of the VM (and it worked! ).

and make sure you have a sane grub.cfg (I’d suggest replacing all references to UUIDs in grub.cfg and /etc/fstab to /dev/vda[1]).
Now, outside the chroot, run:
grub-install --root-directory=/mnt --grub-mkdevicemap=/mnt/boot/grub/device.map /dev/loop0

snf-image-creator takes a diskdump as input, launches a helper VM, cleans up the diskdump / image (cleanup of sensitive data etc), and optionally uploads and registers our image with ~okeanos.

For more information on how snf-image-creator and Synnefo image registry works, visit the relevant docs [1][2][3].

0) Since snf-image-creator will use qemu/kvm to spawn a helper VM, and we’re inside a VM, let’s make sure that nested virtualization (OSDI ’10 Best Paper award btw ) works.

First, we need to make sure that kvm_[amd|intel] is modprobe’d on the host machine / hypervisor with the nested = 1 parameter, and that the vcpu, that qemu/kvm creates, thinks that it has ‘virtual’ virtualization extensions (that’s actually our responsibility, and it’s enabled on the okeanos.io servers).

If everything goes as planned, after snf-image-creator terminates, you should be able to see your newly uploaded image in https://pithos.okeanos.io, inside the Images container. You should also be able to choose your image to create a new VM (either via the Web UI, or using the kamaki client).

That’s all for now. Hopefully, I’ll return soon with another more detailed post on scripting with kamaki (vkoukis has a nice script using kamaki python lib to create from scratch a small MPI cluster on ~okeanos ).

These XO-1.75 is based on the Marvell Armada 610 SoC (armv7l, non-NEON), which promises countless hours of fun enumerating and obtaining obscure pieces of software which are needed to make the laptop work.

One of these is the xf86-video-dove DDX for the Vivante(?) GPU: The most recent version 0.3.5 seems to be available only as SRPM in the OLPC rpmdropbox. Extracting it reveals a "source" tarball containing this:

At the Gentoo Miniconf 2012 in Prague we
will install Gentoo on the OLPC
XO-1.75, an ARM based laptop designed as an educational tool for
children. If you are interested in joining us, come to the Gentoo booth and
start hacking with us!

As the quantity and quality of this year's entries will attest, Gentoo is
alive, well, and taking no prisoners!

We had 70 entries for the 2012 Gentoo screenshot contest, representing 11 different window managers / desktop environments. Thanks to all that participated, the judges and likewhoa for the screenshot site.

In the last few weeks, the conference team has worked hard to prepare the conference. The main news items you should be awere of are the FAQ which has been published, the party locations and times, the call to organize BoF sessions and of course the sponsors who help make the event possible. And we’re happy to tell you that we will provide live video streams from the main rooms during the event (!!!) and we announced the Round Table sessions during the Future Media track. Last but not least, there have been some interviews with intresting speakers in the schedule!

Talking about video interviews, there will be more videos in those channels: the openSUSE Video team is gearing up to tape the talks at the event. They will even provide a live stream of the event, which you can watch via flash and on a smartphone at bambuser and via these three links via ogv feeds: Room KirkRoom McCoy and Room Scotty. Keep an eye on the wiki page as the team will add feeds to more rooms if we can get some more volunteers to help us out.

Despite all our work, this event would be nothing without YOUR help. We’re still looking for volunteers to sign up but there’s another thing we need you for: be pro-active and get the most out of this event! That means not only sitting in the talks but also stepping up and participating in the BoF Sessions. And organize a BoF if you think there’s something to discuss!

Party time!

Of course, we’re also thinking about the social side of the event. Yes, there will surely be an extensive “hallway track” as we feature a nice area with booths and the university has lots of hallways… But sometimes it’s just nice to sit down with someone over a good beer, and this is where our parties come in. As this article explains, there will be two parties: one on Friday, as warming-up (and pre-registration) and one on Saturday, rockin’ in the city center of Prague. Note that you will need your badge to enter this party, which means you have to be registered!

Sponsors

As we wrote a few days ago, all this would not be possible without our sponsors, and we’d like to thank them A LOT for their support!

Big hugs to Platinum Sponsor SUSE, Gold Sponsor Aeroaccess, Silver Sponsor Google, Bronze Sponsor B1Systems, supporters ownCloud and Univention and of course our media partners LinuxMagazine and Root.cz. Last but not least, a big shout-out to the university which is providing this location to us!

FaQ

On a practical level, we also published our Conference FAQ answering a bunch of questions you might have about the event. If you weren’t sure about someting, check it out!

More

There will be more news in the coming days, be sure to keep an eye on news.opensuse.org for articles leading up and of course during the event. As one teaser, we’ve got the Speedy Geeko and Lightning talks schedule coming soon!

Be there!

Gentoo Miniconf, oSC12 and LinuxDays will take place at the Czech Technical University in Prague. The campus is located in the Dejvice district and is next to an underground station that gets you directly to the historic city center – an opportunity you can’t miss!

We expect to welcome about 700 Open Source developers, testers, usability experts, artists and professional attendees to the co-hosted conferences! We work together making one big, smashing event! Admission to the conference is completely free. However for oSC a professional attendee ticket is available that offers some additional benefits.

All the co-hosted conferences will start on October 20th. Gentoo Miniconf and Linuxdays end on October 21st, while the openSUSE Conference ends on October 23rd. See you there!

It’s been a while since I’ve done anything visible to anyone but myself. So, what the heck have I been doing?

Well, for starts, in the past year I’ve done a serious amount of work in Python. This work was one of the reasons for my lack of motivation for Gentoo. I went from doing little programming / maintenance at work to doing it 40+ hours a week. It meant I didn’t really feel up to doing more of it in my limited spare time. So I took up a few new hobbies. I got into Photography (feel free to look under links for the photo website). I feel weird with the self promotion for that type of thing, but, c’est la vie.

As the programming at work died down some, I started to find odd projects. I spent some serious time learning Go [1] and did a few small projects of my own in that. One of those projects will be open sourced soon. I know a fair few different languages, and I know C, Python, and Java pretty decently. While I like all of the ones on that list, I can’t say that I truly buy into the philosophies. Python is great. It’s simple, it’s clean, and it “just works.” However, I find that like OpenSSL, it gives you enough room to hang yourself and everyone else in the room. The lack of strict typing coupled with the fact that it’s a scripting language are downsides (in my eyes). C, for all that it is awesome at low level work, requires so much verbosity to accomplish the simplest tasks that I tend to shy away from it for anything other than what must be done at that level. Java… is well Java. It’s a decent enough language I suppose, but being run in a VM is silly in my eyes. It, like C, suffers from being too verbose as well (again, merely my humble opinion).

Enter Go. Go has duck typed interfaces, unlike Java’s explicit ones. It’s compiled and strictly typed. It has other modern niceties (like proper strings), along with a strong tie to web development (another area C struggles with). It has numerous interesting concepts (check out defer), along with what I find to be a MUCH better approach to error handling than what exists in any of C, Java, or Python. Add in that it is concurrent by design and you have one serious language. I must say that I am thoroughly impressed. Serious Kudos to those Google guys for one awesome language.

I also picked up a Nexus 7 and started looking into how Android is built and works. I got my own custom ROM and Kernel working along with a nice Gentoo image on the SD Card. Can anyone say “Go compiler on my Nexus 7?” This work also led me to do some work as far as getting Gentoo booting on Amazon’s Elastic Compute Cloud. Building Android takes for-freaking-ever, so I figured.. why not do it in the cloud!? It works splendidly, and it is fast.

So that covers new tricks. You mentioned goals and ideas?!

First, time to get myself off the slacker wagon and back to doing something useful. I no longer repulse at the idea of developing when I get home. That helps =p. One of the first things I want to spend some time addressing is disk encryption in Gentoo. I wrote here pertaining to the state of loop-aes. Both Loop-AES and Truecrypt need to spend a little time under the microscope as to how they should be handled within Gentoo. I’ll write more on his later when I have all my ducks in a row. I have no doubt that this will be a fun topic.

I also want to look into how a language like Go fits into Gentoo. Go has it’s own build system (no Makefiles, configure scripts, or anything else) that DOES have a notion of things like CFLAGS. It also has the ability to “go get” a package and install it. To those curious check out their website. All of these lead to interesting questions from a package management point of view. I am inclined to think that Go is around to stay. I hope it is. So we may as well start looking into this now rather than later. As my father used to tell me all the time, “Proper Prior Planning Prevents Piss Poor Performance.” Time to plan =).

That is, right after I sort out the fiasco that is my bug queue. *facepalm*

my main gentoo workstation is down. no more documentation updates from me for awhile.

it seems the desktop computer’s video card has finally bitten the dust. the monitor comes up as “no input detected” despite repeated reboots. so now i’m faced with a decision: throw in a cheap, low-end GFX card as a stopgap measure, or wash my hands of 3 to 6 years of progressive hardware failure, and do a complete rebuild. last time i put anything new in the box was probably back in 2009…said (dead) GFX card, and a side/downgraded AMD CPU. might be worth building an entirely new machine from scratch at this point.

i haven’t bothered to pay attention to the AMD-vs-Intel race for the last few years, so i’m a bit at a loss. i’ll check TechReport, SPCR, NewEgg, and all those sites, but…not being at all caught up on the bang-for-buck parts…is a bit disconcerting. i used to follow the latest trends and reviews like a true technoweenie.

and now, of course, i’m thinking in terms of what hardware lends itself to music production — USB/Firewire ports, bus latency, linux driver status for crucial bits; things like that. all very challenging to juggle after being out of it for so long.

Not that long ago we had our monthly Gentoo Hardened project meeting (on October 3rd to be exact). On these meetings, we discuss the progress of the project since the last meeting.

For our toolchain domain, Zorry reported that the PIE patchset is updated for GCC, fixing bug #436924. Blueness also mentioned that he will most likely create a separate subproject for the alternative hardened systems (such as mips and arm). This is mostly for management reasons (as the information is currently scattered throughout the Gentoo project at large).

For the kernel domain, since version 3.5.4-r2 (and higher), the kernexec and uderef settings (for grSecurity) should no longer impact performance on virtualized platforms (when hardware acceleration is used of course), something that has been bothering Intel-based systems for quite some time already. Also, the problem with guest systems immediately reserving (committing) all memory on the host should be fixed with recent kernels as well. Of course, this is only true as long as you don’t sanitize your memory, otherwise all memory gets allocated regardless.

In the SELinux subproject, we now have live ebuilds allowing users to pull in the latest policy changes directly from the git repository where we keep our policy at. Also, we will see a high commit frequency in the next few weeks (or perhaps even months) as Fedora’s changes are being merged with upstream. Another change is that our patchbundles no longer contain all individual patches, but a merged patch. This increases the deployment time of a SELinux policy package considerably (up to 30% faster since patching is now only a second or less). And finally, the latest userspace utilities are in the hardened-dev overlay ready for broader testing.

grSecurity is still focusing on the XATTR-based PaX flags. The eclass (pax-utils) has been updated, and we will now be looking at supporting the PaX extended attributes for file systems such as tmpfs.

For profiles, people will notice that in the next few weeks, we will be dropping the (extremely) old SELinux profiles as the current ones have been marked stable long time ago.

In the system integrity domain, IMA is being worked on (packages and documentation) after which we’ll move to the EVM support to protect extended attributes.

And finally, klondike held a good talk about Gentoo Hardened at the Flossk conference in Kosovo.

All in all a good month of work, again with many thanks to the volunteers that are keeping Gentoo Hardened alive and kicking!