Welcome to Gentoo Universe, an aggregation of weblog articles on all topics written by Gentoo developers. For a more refined aggregation of Gentoo-related topics only, you might be interested in Planet Gentoo.

Okay so last time I wrote about my personal status I noted that I had something on the balance, as a new job. Now that I signed the contract I can say that I do have a new job.

This means among other things that I’ll finally be leaving Italy. My new home is going to be Dublin, Ireland. At the time of writing I’m still fretting about stuff I need to finish in Italy, in particular digitizing as many documents as possible so that my mother can search through them easily, and I can reach them if needed, contacting my doctor for a whole blood panel, and the accountant to get all the taxes straightened up.

What does this mean for my Gentoo involvement? Probably quite a bit. My new job does not involve Gentoo, which means I won’t be maintaining it any longer on paid time like I used to before. You can also probably guess that with the stress of actually having a house to take care of, I’ll end up with much less time than I have now. Which means I’ll have to scale down my involvement considerably. My GSoC project might very well be the height of my involvement from now till the end of the year.

On the personal side of things, while I’m elated to leave Italy, especially with the current political climate, I’m also quite a bit scared. I know next to nobody (Enrico excluded) in Dublin, and I know very little of Irish traditions as well. I’ve spent the past week or so reading the Irish Times just to be able to catch a glimpse of what is being discussed up there, but I’m pretty sure that’s not going to be enough.

I’m scared also because this would be the first time I actually leave alone and have to cater for everything by myself, even though with the situation it feels like I might be quite a lot more lucky than most of my peers here in Clownland Italy. I have no idea of what will actually sap away my time, although I’m pretty sure that if it turns out to be cleaning, I’ll just pay somebody to do that for me.

One question that I’ve been asked before, and to which I didn’t really have a good answer up to now is: should configure scripts fail, when a dependency is enabled explicitly, but it can’t be found? This is the automagic dependency problem, but on the other branch.

With proper automatic dependencies, if the user does not request explicitly whether to enable something or not, it’s customary that the dependency is checked for, and if found, the feature that it’s connected to is enabled. When the user has no way to opt out from it (which is bad), we call it an automagic dependency. But what happens if the user has requested it and the feature is not available?

Unfortunately, there is no standard for this, and myself I used both the “fail if asked and not found” and “warn if asked and not found” approaches. But the recent trouble between ncurses and freetype made me think that it’s important to actually make a point that there is a correct way to deal with this.

Indeed what happens is that right now, I have no way to tell you all that the tinderbox has found every single failure caused by sys-libs/ncurses[tinfo] even after the whole build completed: it might well be that a particular package, unable to link to ncurses, decided to disable it altogether. The same goes for freetype. Checking for all of that would be nice, but I have honestly no way to do it.

So to make sure that the user gets really what they want, please always make sure that you do verify that you’re proceeding how the user wanted. This makes sure that even in packaging, there won’t be any difference when a dependency is updated, or changed. In particular, with pkg-config, the kind of setup you should have is the following:

I’ll be discussing this and proposing this solution in the next update to Autotools Mythbuster (which is due anytime soon, including the usual eBook update for Kindle users). This would hopefully make sure that in the future, most configure scripts will follow this approach.

There's a lot of people who are very careful to never delete a single line from an e-mail they are replying to,
always quoting the complete history. There's also a lot of people who believe that it wastes time to eyeball such long,
useless texts. One of the fancy features introduced in this release of Trojitá,
a fast Qt IMAP e-mail client, is automatic quote collapsing. I won't show you an example of an annoying mail for obvious
reasons :), but this feature is useful even for e-mails which employ reasonable quoting strategy. It looks like this in
the action:

When you click on the ... symbols, the first level expands to reveal the following:

When everything is expanded, the end results looks like this:

This concept is extremely effective especially when communicating with a top-posting community.

We had quite some internal discussion about how to implement this feature. For those not familiar with Trojitá's
architecture, we use a properly restricted QtWebKit instance for e-mail rendering. The restrictions which are active
include click-wrapped loading of remote content for privacy (so that a spammer cannot know whether you have read their
message), no plugins, no HTML5 local storage, and also no JavaScript. With JavaScript, it would be easy to do
nice, click-controlled interactive collapsing of nested citations. However, enabling JavaScript might have quite some
security implications (or maybe "only" keeping your CPU busy and draining your battery by a malicious third party). We
could have enabled JavaScript for plaintext contents only, but that would not be as elegant as the solution we
chose in the end.

Starting with Qt 4.8, WebKit ships with support for the :checked CSS3 pseudoclass. Using this feature,
it's possible to change the style based on whether an HTML checkbox is
checked or not . In theory, that's everything one might possibly need, but there's a small catch
-- the usual way of showing/hiding contents based on a state of a checkbox hits a WebKit bug (quick summary: it's tough to have it working without the
~ adjacent-sibling selector unless you use it in one particular way). Long story short, I now know more
about CSS3 than I thought I would ever want to know, and it works (unless you're on Qt5 already where
it assert-fails and crashes the WebKit).

Speaking of WebKit, the way we use it in Trojitá is a bit unusual. The QWebView class contains full
support for scrolling, so it is not necessary to put it inside a QScrollArea. However, when working with
e-mails, one has to account for messages containing multiple body parts which have to be shown separately (again, for
both practical and security reasons). In addition, the e-mail header which is typically implemented as a custom
QWidget for flexibility, is usually intended to combine with the message bodies into a single entity to be
scrolled together. With WebKit, this is doable (after some size hints magic, and I really mean magic -- thanks
to Thomas Lübking of the KWin fame for patches), but there's a catch -- internal methods like the findText
which normally scroll the contents of the web page into the matching place no longer works when the whole web view is
embedded into a QScrollArea. I've dived into the source code of WebKit and the interesting thing is that there
is code for exactly this case, but it is only implemented in Apple's version of WebKit. The source code even says that Apple needed this for its own
Mail.app -- an interesting coincidence, I guess.

Compared with the last release, Trojitá has also gained support for "smart replying". It will now detect that a
message comes from a mailing list and Ctrl+R will by default reply to list. Thomas has added support for
saving drafts, so that you are not supposed to lose your work when you accidentally kill Trojitá anymore. There's also
been the traditional round of bug fixes and compatibility improvements. It is entertaining to see that Trojitá is
apparently triggering certain code paths in various IMAP server implementations, proprietary and free software alike,
for the first time.

The work on support for multiple IMAP accounts is getting closer to being ready for prime time. It isn't present in
the current release, though -- the GUI integration in particular needs some polishing before it hits the masses.

I'm happy to observe that Trojitá is getting features which are missing from other popular e-mail clients. I'm
especially fond of my pet contribution, the quote collapsing. Does your favorite e-mail application offer a similar
feature?

In the coming weeks, I'd like to focus on getting the multiaccounts branch merged into master, adding better
integration with the address book (Trojitá can already offer tab completion with data coming from Mutt's abook) and general GUI improvements. It would also be great to make it possible
to let Trojitá act as a handler for the mailto: URLs so that it gets invoked when you click on an e-mail
address in your favorite web browser, for example.

And finally, to maybe lure a reader or two into trying Trojitá, here's a short quote from a happy user who came to
our IRC channel a few days ago:

17:16 < Sir_Herrbatka> i had no idea that it's possible for mail client to be THAT fast

One cannot help but be happy when reading this. Thanks!

If you're on Linux, you can get the latest version of Trojitá from the OBS or the usual place.

I said this last week on Google+ when I was at a conference, and
needed to get it out there quickly, but as I keep getting emails and
other queries about this, I might as make it "official" here. For no
other reason that it provides a single place for me to point people at.

Anyway, I would like to announce that the 3.8 Linux kernel series is
NOT going to be a longterm stable kernel release. I will
NOT be maintaining it for long time, and in fact, will stop
maintaining it right after the 3.9 kernel is released.

The 3.0 and 3.4 kernel releases are both longterm, and both are going to
be maintained by me for at least 2 years. If I were to pick 3.8 right
now, that would mean I would be maintaining 3 longterm kernels, plus
whatever "normal" stable kernels are happening at that time. That is
something that I can not do without loosing even more hair than I
currently have. To do so would be insane to attempt.

In my previous post I didn’t add much about Gentoo ideas for GSoC and I didn’t really volunteer on anything. Well, this changes now as I have a suggestion and I’m even ready to propose myself as a mentor — with the caveat that as every other year (literally) my time schedule is not clear, so I will need a backup co-mentor.

You might or might not remember, but for my tinderbox I’m using a funky log collection and analysis toolset. Said toolset, although completely lacking chewing gum, is not exactly an example of good engineering, but it’s really just a bunch of hacks that happen to work. On the other hand, since doing Gentoo tinderboxing has never been part of my job, I haven’t had the motivation to rewrite it to something decent.

While the (often proposed) “continuous integration solution” for Gentoo is a task that is in my opinion unsuitable for a GSoC student – which should be attested by the fact that we don’t have one yet! – I think that at least the collection-and-analysis part should be feasible. I’ll try to list and explain here the current design and how I’d like it to work, so if somebody feels like working on this, they can already look into what there is to do.

Yes, I know that Zorry and others involved in Gentoo Hardened have been working on something along these line for a while — I still haven’t seen results, so I’m going to ignore it altogether.

Right now what happens is that we have four actors (in sense of computer/systems) involved: the tinderbox itself, a collector, a storage system, and finally, a frontend.

Between the tinderbox and the collector, the only protocol is tar-over-tcp, thanks to, well, tar and netcat on one side, and Ruby on the other. Indeed, my tinderbox script sends (encapsulated in tar) every completed log to the collector, which then extracts the log, and parses it.

The collector does most of the heavy lifting right now: it gets the package name and the timestamp of the log from the name of the file in the tar archive, then the maintainers (to assign the bug) from the log itself. It scans for particular lines (Portage and custom warning, among others), and creates a record with all of that together, to be sent to Amazon’s SimpleDB for querying. The log file itself is converted to HTML, split line by line, so that it can be seen with a browser without downloading, and saved to Amazon’s S3 once again.

The frontend fetches the records from SimpleDB where at least one warning is to be found, and display them in a table, with a link to see the log, and one to open a bug (thanks to pre-filled templates). It’s implemented in Sinatra as it is, and it’s definitely simplistic.

Now there are quite a number of weak spots in this whole setup. The most obvious is the reliance on Amazon. It’s not just an ethical or theoretical question, but the fact that S3 makes you pay for per-access is the reason why the list of logs is not visible to the public right now (it would easily add up to costs quickly).

What I would like from a GSoC project would be a replacement for the whole thing in which there can be a single entity that covers the collector, storage and frontend, without relying on Amazon at all. Without going all out on features that are difficult to manage, you need to find a way to store big binary data (the logs themselves can easily become many gigabyte in size), and then have a standard database with the list of entries like I have now. Technically, it would be possible to keep using S3 for logs, but I’m not sure how much of a good idea that would be at this point.

Of course, one of the reasons why the collector and the frontend are split in my current setup, is that I thought that network connectivity between the tinderbox and the Internet couldn’t be entirely guaranteed; on the other hand, while the connection between the tinderbox and the collector is guaranteed (they are on the same physical host), the collector might not have an Internet connection to push to S3/SimpleDB, so…

On the frontend side, a new frontend would have a better integration with Bugzilla, for instance it would be nice if I could just open the log and have a top frame (or div, I don’t care how it’s implemented) showing me a quick list of bugs open for the package, so I no longer have to search for it on Bugzilla to see if the problem has been reported already or not. Would also be nice to be able to attach the log to the newly open bugs, but that’s something that I care for relatively, if nothing else because some logs are so big, that to attach them you’d have to compress them with xz, and even then they might not work.

But possibly the most important part of this would be to change the way the tinderbox connects to the collector. Instead of keeping tar and netcat, I would like for Portage to implement a “client” by itself. This would make it easier to deploy the collector for uses different from tinderboxing (such as an organization-wide deployment), and at the same time would allow (time permitting) expansion on what is actually sent. For instance right now I have to gather and append the autoconf, automake and similar error logs to the main build log, to make sure that we can read it when we check the build log on the bug… if Portage was able to submit the logs by itself, it would submit the failure logs as well as config.log and possibly other output file.

Final note: no you can’t store the logs in an SQL database. But if you’re going to take part in GSoC and want to help with the tinderbox, this would be what I (as the tinderbox runner) need the most, so … think about it! I can provide a few more insights and the why of them (for instance why I think this really got to be written in Python, and why it has to support IPv6), if there is the interest.

It’s no mystery I hated Boost’s slotted package handling, and that’s captured by the fact that I unslotted it last November as it was doing nothing good for us. But one of the most common comparisons at that point is with Berkeley DB (sys-libs/db) which we still have slotted and I have no intention to change any time soon. So let me document a bit the reason behind this, as they tie up quite nicely with some other tasks connected to berkdb that I’ll get to at the end of the post.

First of all, whether you like or not the non-exactly-clinical assessment, Berkeley DB APIs have been much more stable than Boost’s — even the bump to 5.0 hasn’t been as bad as many thinks, the biggest problem I’ve noticed with it is a broken version check logic:

Sometimes it goes even worse and instead of checking for a major of exactly 4, it checks for major greater or equal to 4 and minor greater or equal to X — which ends up working on 5.x but not on 5.(x-1). The fixes for these are trivial and usually are more a nuisance than a real problem.

But this is only part of the reason. While the API is stable, the ABI really is not (it’s possible to change API in such a way that it produces a different ABI, but it requires no change to the source code), which means that we do need to rebuild anything built against the older version, after an update. This is not so uncommon though, for unslotted packages as well, so it can’t be enough to warrant the slotting.

Here is the all-important detail: Berkeley DB mandates a file format, and that file format changes between versions! Okay it does not always change, but it changes often enough. What happens in this situation is that the files created by Berkeley DB version X will not be read by version Y and vice-versa — this is not much of a problem when the DB files are used as cache for fast access – like Postfix maps or, in my case, Apache rewrite maps – but it is when you use them for actually storing information that can’t be regenerated directly, and there pam_userdb comes to mind.

Indeed, for a long time, pam_userdb was built against a single copy of Berkeley DB, rather than using the system one ­— this way it would not change file format between rebuilds. Unfortunately this also had the effect of skipping over security updates, which was not considered very good. The current PAM ebuilds can build the module against system Berkeley DB, as the rare situations where these modules are used do not warrant the extra work to maintain the bundled copy of it.

So while this gets us near to the other topic I want to talk about in this post, how does slotting the library get us? The answer is that we don’t just slot the library, we also slot the tools that come with it, including the dump and restore tools. What happens then is that you can dump with the old slotted tool the content of the file that you can’t recreate, and restore it with the new one (the new tools read the dump of the older ones). By slotting the package, you’ll always have the set of tools that read the older databases.

This is the reason why we have a slotted Berkeley DB.

Now to complete the post I promised something else. Well, you probably have noticed because of the fallout, but there is work underway for a realistic native multilib eclass, that allows us to build in a single pass both 64- and 32-bit version of a single package. Why is this relevant? Well, if you got a 32-bit proprietary service that uses PAM, you need a 32-bit PAM library and 32-bit PAM modules as well…

If we were to provide pam_userdb with the emul-linux packages (which we do!) it’s very well possible that the two versions of Berkeley DB might not match on a given system, because the user is using a different one altogether. Being able to use a single ebuild for both ABIs would make much more sense, and is definitely something I’m looking forward to!

So from time to time I like to write down what my tinderbox is doing and this sounds like the perfect time to do so.

First of all, GCC 4.7 is getting unmasked soonish — the number of new bugs in the tracker is basically zero (I opened one today for a newly bumped package), and since at least a little bit of newly fixed bugs appear every other week, it should soon be feasible to just use GCC 4.7.

But the important part is the new tasks; both the ~amd64 and the stable/hardened amd64 tinderboxes are currently running aimed at two different packages’ reverse dependencies.

The former is running against the new, multilib-capable media-libs/freetype package, which Michał introduced recently — this version is able to build the 32-bit together with the 64-bit one, making the binary emul-linux package unnecessary for binary, 32-bit software requiring it. Finally being able to configure which 32-bit packages to install is going to be a win especially for systems with reduced storage space. Unfortunately, Freetype has a design that I would call naïve not to fall into foul language, and its headers are actually architecture-dependent; this means that the 64-bit and the 32-bit builds have different header files. Can you tell how bad that is?

While the proper solution would be to fix the design so that it does not have different headers at all, what we can do in the mean time is installing both sets in different paths, so that they do not collide with each other and software does not build against the wrong version. Unfortunately, since this means moving away even the native ABI headers, it means that you need to tell every single package where to find them. This should be solved by using pkg-config but unfortunately a number of packages out there don’t do that, especially non-autotools based ones — CMake packages seem to use pkg-config … but then they expect to guess the position of the headers instead of actually accepting what pkg-config says.

So I’m now building the reverse dependencies (at least those that the tinderbox can build, some of the games have interactive properties, because they require a CD to be mounted, and sometimes the dependency chain breaks too soon), and I’m going to report them in the tracker bug so that they can be tackled.

The second task is a bit more important from a point of view (even though it’s easier to not trigger): sys-libs/ncurses gained recently an USE flag tinfo that makes it build a separate library for some of the symbols that were, before, collapsed into libncurses alone. Unfortunately this means that now the symbols are defined in libtinfo and nobody is looking for it (or very few). The solution is, for this as well, to rely on what pkg-config tells you, but most packages do not use it for ncurses, mostly for compatibility with very old ncurses versions as far as I can tell.

It wouldn’t be half-bad if the same guy who ignore my bug reports, and who unmask glibc versions that are known to break half the tree didn’t decide to introduce said USE flag into stable already, knowing full well that most of the tree hasn’t been tested in that particular configuration. And here’s where my tinderbox enter the scenes, as it’s currently checking the stable tree to make sure I can report all packages breaking when that USE flag is enabled.

So the builds are running and the bugs are flowing — please just remember that the first primary resource that the tinderbox consumes is my time.

You might remember that a couple of years ago, I bought an EPSON scanner – a GT-S50, no longer sold, replaced by the GT-S55& – to scan invoices and other kind of documents that I had laying around. The investment paid off, in my eyes, because now I have all my documents digitized, always available, and I was able to fill quite a few bags of shredded paper. The amount of work involved in digitizing that many documents without a professional scanner would have costed me in time much more than the device costed me in money.

But as I noted in the blog post above, to get the scanner to work on Gentoo Linux, I had to create ebuilds for a new package with the proprietary plugin that can handle the scanner — it’s not a sane plugin per-se, it’s rather a plugin for the EPSON-provided epkowa backend, which is otherwise opensource. I actually ended up adding two packages to the tree to be able to do that.

Nowadays, the media-gfx category hosts a number of plugins for epkowa, under iscan-plugin and esci-interpreter prefixes — the latter is the case for the GT-S50; as far as I can tell, the main difference between the two is that the latter ships with no firmware, but just a protocol interpreter, as the name suggests. Together with this, I maintain another ebuild for the plugin (32-bit only) and firmware (32- and 64-bit) for my older Perfection 2580 scanner, plus two more packages that I proxy for users who can actually test on good hardware. Today I even added another package for a scanner I don’t own, but I plan to buy when I settle down in my new residency (the 2580 stays here with my mother), the Perfection V370& which is the first one in our tree that actually uses “perfection” in the RPM name.

The ebuilds are almost identical to one another, but there are quite a few tricky bits around them. First of all, we don’t install in exactly the same locations as the RPMs — the reason for this is that they use /usr as base path for everything, but in our case, since these are provided by an external source, and are pre-built, they are installed in /opt instead, with the exception of the firmware, that stays in /usr/share simply because that’s where some of the software looks for it (namely, snapscan backend for SANE).

Some of the plugins are valid for multiple USB IDs, as is the case for the GT-S50/55/80/85 interpreter, while others are only valid for a particular ID – which may refer to multiple models anyway, as sometimes it refers to the main controller on the device, which may be shared — in the case of the 2580 noted above, it shares the same flatbed with the 2480; the difference between the two models is all in the lid, in both it hosts the lamp for scanning film, but in the 2580 it also includes the motor to load the film semi-automatically.

You can easily see which IDs the plugin applies to by using strings on the rpm and look for iscan-registry — but at this point question should be: where do you get the RPM? Well, unfortunately AVASYS no longer provides the download at a known location; EPSON themselves are distributing it, through Akamai — which means that we can’t point to the upstream servers anymore, and euscan won’t be able to find a new version being released. To solve this situation, the solution we opted for is to mirror the rpm files on my webspace on dev.gentoo.org including the needed documentation. It’s not very flexible, of course, but at least it works and it does not require us to fetch-restrict anything.

It’s hard for me to tell how many more plugins for iscan are out there, but if your scanner is among them, feel free to send an ebuild for it on our Bugzilla and CC me — you can base yourself off media-gfx/iscan-plugin-perfection-v370 which is the latest I committed.

My work on Ruby-Elf tends to happen in “sprees” whenever I actually need something from it that wasn’t supported before — I guess this is true for many projects out there, but it seems to happen pretty regularly for me with my projects. The other day I prepared a new release after fixing the bug I found while doing the postmortem of a libav patch — and then I proceeded giving another run to my usual collisions check after noting that I could improve the performance of the regular expressions …

But where is it directed, as it is? Well, I hope I’ll be able to have version 2.0 out before end of 2013 — in this version, I want to make sure I get full support for archives, so that I can actually analyze static archives without having to extract them beforehand. I’ve got a branch with the code to get access to the archives themselves, but it can only extract the file before actually being able to read it. The key in supporting archives would probably be supporting in-memory IO objects, as well as offset-in-file objects.

I’ve also found an interesting gem called bindata which seems to provide a decent way to decode binary data in Ruby without having to actually fully pre-decode it. This would probably be a killer for Ruby-Elf, as a lot of the time I’m forcibly decoding everything because it was extremely difficult to access it on the spot — so the first big change for Ruby-Elf 2 is going to be to drop down the task of decoding to bindata (or, otherwise, another similar gem).

Another change that I plan is to drop the current version of the man pages. While DocBook is a decent way to deal with man pages, and standard enough to be around in most distributions, it’s one “strange” dependency for a Ruby package — and honestly the XML is a bit too verbose sometimes. For the most horsey beefy man pages, the generated roff page is half as big as the source, which is the other way around from what anybody would expect them.

So I’m quite decided that the next version of Ruby-Elf will use Markdown for the man pages — while it does not have the same amount of semantic tagging, and thus I might have to handle some styling in the synopsis manually, using something like md2man should be easy (I’m not going to use ronn because of the old issue with JRuby and rdiscount) and at the same time, it gives me a public HTML version for free, thanks to GitHub conversion.

Finally, I really hope that by Ruby-Elf 2 I’ll be able to get least the symbol demangler for the Itanium C++ ABI — that is the one used by modern GCC, yes, it was originally specified for the Itanic. Working toward supporting the full DWARF specification is something that is on the back of my mind but I’m not very convinced right now, because it’s huge. Also, if I were to implement it I would then have to rename the library to Dungeon.

At the time of writing (but I’ll delay the publication of this post a few hours), I’m uploading a new SELinux-enabled KVM guest image. This is not an update on the previous image though (it’s a reinstalled system – after all, I use VMs for testing, so it makes sense to reinstall from time to time to check if the installation instructions are still accurate). However, the focus remains the same:

A minimal Gentoo Linux installation for amd64 (x86_64) as guest within a KVM hypervisor. The image is about 190 Mb in size compressed, and 1.6 Gb in size uncompressed. The file format is Qemu’s QCOW2 so expect the image to grow as you work with it. The file systems are, in total, sized to about 50 Gb.

The installation has SELinux enabled (strict policy, enforcing mode), various grSecurity settings enabled (including PaX and TPE), but now also includes IMA (Integrity Measurement Architecture) and EVM (Extended Verification Module) although EVM is by default started in fix mode.

The image will not start any network-facing daemons by default (unlike the previous image) for security reasons (if I let this image stay around this long as I did with the previous, it’s prone to have some vulnerabilities in the future, although I’m hoping I can update the image more frequently). This includes SSH, so you’ll need access to the image console first after which you can configure the network and start SSH (run_init rc-service sshd start does the trick).

A couple of default accounts are created, and the image will display those accounts and their passwords on the screen (it is a test/play VM, not a production VM).

There are still a few minor issues with it, that I hope to fix by the next upload:

Bug 457812 is still applicable to the image, so you’ll notice lots of SELinux denials on the mknod capability. They seem to be cosmetic though.

At shutdown, udev somewhere fails with a SELinux initial context problem. I thought I had it covered, but I noticed after compressing the image that it is still there. I’ll fix it – I promise ;)

EVM is enabled in fix mode, because otherwise EVM is prohibiting mode changes on files in /run. I still have to investigate this further though – I had to use the EVM=fix workaround due to time pressure.

When uploaded, I’ll ask the Gentoo infrastructure team to synchronise the image with our mirrors so you can enjoy it. It’ll be on the distfiles, under experimental/amd64/qemu-selinux (it has the 20130224 date in the name, so you can see for yourself if the sync has already occurred or not).

I’ve got to admit that the whole “Test-Driven Development” hype was not something that appealed to me as much — not because I think tests are wrong, I just think that while tests are important, focusing almost exclusively on them is just as dumb as ignoring them altogether.

In the Ruby world, there is so much talking about tests, that it’s still very common to find gems that don’t work at all with newer Ruby versions, but their specs pass just fine. Or even the tests pass fine for the original author, but they will fail on everyone else’s system because they depend heavily on custom configuration — sometimes, they depend on case-insensitive filesystems because the gem was developed on Windows or Mac, and never tested on Linux. Indeed, for the longest time, Rails own tests failed to work at all chances, and the “vendored” code they brought in, never had a working testsuite. Things have improved nowadays but not significantly.

Indeed, RubyGems do not make it easy to perform testing upon install, which means that many gems distributed lack part of the testsuite altogether ­— sometimes this is an explicit choice; in the case of my own RubyElf gem the tests are not distributed because they grow and grow, and they are quite a bit of megabytes at this point; if you want to run them you fetch the equivalent snapshot from GitHub — the ebuild in Gentoo uses that as a basis for that reason.

Sometimes even gems coming from people who are considered nearly Ruby Gods, like "rails_autolink"https://rubygems.org/gems/rails_autolink by tenderlove end up with a gem that fails tests, badly, in its release — the version we have in Portage is patched up, and the patch is sent upstream. Only the best for our users.

Now unfortunately, as I noted in the post’s title, some projects care more about the tests than the functionality — the project in question is the very same Typo that I use for this blog, and which I already mused forking to implement fixes that are not important for upstream. Maybe I should have done that already, maybe I will do that.

So I sent a batch of changes and fixes to upstream, some of them fixing issues compelled by their own changes, other implementing changes to allow proper usage of Typo over SSL vhosts (yes my blog is now available over SSL — I have to fix a few links and object load paths in some of the older posts, but it will soon work fine), other again simply making it a bit more “SEO”-friendly, since that seems to be a big deal for the developers.

What kind of response do I get about the changes? “They fail spec” — no matter that the one commit I’m first told it breaks specs actually fix editing of blog post after a change that went straight to master, so it might break specs, but it solve a real life issue that makes the software quite obnoxious. So why did I not check specs?

I have no intention to start looking into this whole set of gems just to be able to run the specs for a blog which I find are vastly messed up. Why do I think so? Well, among other reasons, I’ve been told before quite a few times that they wouldn’t ever pass on PostgreSQL — which happens to be the database that has been powering this very instance for the past eight years. I’m pretty sure it’s working good enough!

Well, after asking (a few times) for the specs output — turns out that most of the specs broken are actually those that hardcode http:// in the URLs. Of course they break! My changes use protocol-relative URIs which means that the output changes to use // — no spec is present that tries to validate the output for SSL-delivered blogs which would otherwise break before.

And what is the upstream’s response to my changes? “It breaks here and there, would you mind looking into it?” Nope! “The commit breaks specs.” — No offer (until I complained loudly enough on IRC) for them to look into it, and fix either the patches or the specs are needed. No suggestion that there might be something to change in the specs.

Not even a cherry-pick of the patches that do not break specs.

Indeed as of this writing, even the first patch in the series, the only one that I really would care about get merged, because I don’t want to get out-of-sync with master’s database, at least until I decide to just get to the fork, is still there lingering, even if there is no way in this world that it breaks specs as it introduces new code altogether.

Am I going to submit a new set of commits with at least the visible specs’ failures fixed? Not sure — I really could care more about it, since right now my blog is working, it has the feature, the only one missing being the user agent forwarding to Akismet. I don’t see friendliness coming from upstream, and I keep thinking that a fork might be the best option at this point, especially when, suggesting the use of Themes for Rails to replace the currently theme handling, so that it works properly with the assets pipeline (one of the best features of Rails 3), the answer was “it’s not in our targets” — well it would be in mine, if I had the time! Mostly because being able to use SCSS would make it easier to share the stylesheets with my website (even though I’m considering getting rid of my website altogether).

So my plead to the rest of the development community, which I hope can be considered part of, is to not be so myopic that you care more about tests passing than features working. For sure Windows didn’t reach its popularity level being completely crash-proof — and at the same time I’m pretty sure that they did at least a level of basic testing on it. The trick is always in the compromise, not on the absolute care or negligence for tests.

A long time ago, I made a SELinux enabled VM for people to play with, displaying a minimal Gentoo installation, including the hardening features it supports (PIE/PIC toolchain, grSecurity, PaX and SELinux). I’m currently trying to create a new one, which also includes IMA/EVM, but it looks like I still have many things to investigate further…

First of all, I notice that many SELinux domains want to use the mknod capability, even for domains of which I have no idea whatsoever why they need it. I don’t notice any downsides though, and running in permissive mode doesn’t change the domain behavior. But still, I’m reluctant to mark them dontaudit as long as I’m not 100% sure.

Second, the gettys (I think it is the getty) result in a “Cannot change SELinux context: permission denied” error, even though everything is running in the right SELinux context. I still have to confirm if it really is the getty process or something else (the last run I had the impression it was a udev-related process). But there are no denials and no SELinux errors in the logs.

Third, during shutdown, many domains have problems accessing their PID files in /var/run (which is a link to /run). I most likely need to allow read privileges on all domains that have access to var_run_t towards the var_t symlinks. It isn’t a problem per se (the processes still run correctly) but ugly as hell, and if you introduce monitoring it’ll go haywire (as no PID files were either found, or were stale).

Also, EVM is giving me a hard time, not allowing me to change mode and ownership in files on /var/run. I have received some feedback from the IMA user list on this so it is still very much a work-in-progress.

Finally, the first attempt to generate a new VM resulted in a download of 817 MB (instead of the 158 MB of the previous release), so I still have to correct my USE flags and doublecheck the installed applications. Anyway, definitely to be continued. Too bad time is a scarce resource :-(

So I’ve decided to dust off my link collision script and see what the situation is nowadays. I’ve made sure that all the suppression file use non-capturing groups on the regular expressions – as that should improve the performance of the regexp matching – make it more resilient to issues within the files (metasploit ELF files are barely valid), and run it through.

Well, it turns out that the situation is bleaker than ever. Beside the obvious amount of symbols with a too-common name, there are still a lot of libraries and programs exporting default bison/flex symbols the same way I found them in 2008:

Note that at least one library got to export them to be listed in this output; indeed these symbols are present in quite a long list of libraries. I’m not going to track down each and every of them though, but I guess I’ll keep an eye on that list so that if problems arise that can easily be tracked down to this kind of collisions.

Action Item: I guess my next post is going to be a quick way to handle building flex/bison sources without exposing these symbols, for both programs and libraries.

But this is not the only issue — I’ve already mentioned a long time ago that a single installed system already brings in a huge number of redundant hashing functions; on the tinderbox as it was when I scanned it, there were 57 md5_init functions (and this without counting different function names!). Some of this I’m sure boils down to gnulib making it available, and the fact that, unlike the BSD libraries, GLIBC does not have public hashing functions — using libcrypto is not an option for many people.

Action item: I’m not very big of benchmarks myself, never understood the proper way to go around getting the real data rather than being fooled by the scheduler. Somebody who’s more apt at that might want to gather a bunch of libraries providing MD5/SHA1/SHA256 hashing interfaces, and produce some graphs that can let us know whether it’s time to switch to libgcrypt, or nettle, or whatever else that provides us with good performance as well as with a widely-compatible license.

The presence of duplicates of memory-management symbols such as malloc and company is not that big of a deal, at first sight. After all, we have a bunch of wrappers that use interposing to account for memory usage, plus another bunch to provide alternative allocation strategies that should be faster depending on the way you use your memory. The whole thing is not bad by itself, but when you get one of graphviz’s libraries (libgvpr) to expose malloc something sounds wrong. Indeed, if even after updating my suppression filter to ignore the duplicates coming from gperftools and TBB, I get 40 copies of realloc() something sounds extremely wrong:

Now it is true that it’s possible, depending on the usage patterns, to achieve a much better allocation strategy than the default coming from GLIBC — on the other hand, I’m also pretty sure that GLIBC’s own allocation improved a lot in the past few years so I’d rather use the standard allocation than a custom one that is five or more years old. Again this could use some working around.

In the list above, Thunderbird and Firefox for sure use (and for whatever reason re-expose) jemalloc; I have no idea if libhoard in OpenFOAM is another memory management library (and whether OpenFOAM is bundling it or not), and Mercury is so messed up that I don’t want to ask myself what it’s doing there. There are though a bunch of standalone programs listed as well.

Action item: go through the standalone programs exposing the memory interfaces — some of them are likely to bundle one of the already-present memory libraries, so just make them use the system copy of it (so that improvements in the library trickle down to the program), for those that use custom strategies, consider making them optional, as I’d expect most not to be very useful to begin with.

There is another set of functions that are similar to the memory management functions, which is usually brought in by gnulib; these are convenience wrappers that do error checking over the standard functions — they are xmalloc and friends. A quick check, shows that these are exposed a bit too often:

In this case they are exposed even by the GCC tools themselves! While this brings me again to complain that gnulib show now actually be libgnucompat and be dynamically linked, there is little we can do about these in programs — but the symbols should not creep in system libraries (mandb has the symbols in its private library which is marginally better).

Action item: check the libraries exposing the gnulib symbols, and make them expose only their proper interface, rather than every single symbol they come up with.

I suppose that this is already quite a bit of data for a single blog post — if you want a copy of the symbols’ index to start working on some of the action items I listed, just contact me and I’ll send it to you, it’s a big too big to just have it published as is.

I’ve noted when I posted suggestions for GSoC that we’ve been working on improving the DVD-related libraries that are currently used by most of the open-source DVD players out there: libdvdread and libdvdnav — together with these, me and the Other Diego have been dusting off libdvdcss as well, which takes care of, well, cracking the CSS protection on DVDs so that you can watch your legally owned DVDs on Linux and other operating systems without going crazy.

Yes I did take the picture just to remind you all that I do pay for content so if you find me taking about libdvdcss is not because I’m a piracy apologist because I’m cheap — whenever I do resolve to piracy it’s because it’s nigh impossible to get the content legally, like for what concerns J-Drama.

Anyway, the work we’ve been pouring into these libraries will hopefully soon come into fruition; on my part it’s mostly a build system cleanup task: while the first fork, on mplayer, was trying to replace autotools with a generic, FFmpeg-inspired build system, the results have been abysmal enough that we decided to get back to autotools (I mean, with me on board, are you surprised?) so now they have a modern, non-recursive, autotools based build system. Diego and J-B have been cleaning up the code itself from the conditionals for Windows, and and Rafaël has now started cleaning up libdvdnav’s code by itself.

One of the interesting part of all this is that the symbol table exposed by the libraries does not really match what is exposed by the headers themselves. You can easily find this by using exuberant-ctags – part of dev-util/ctags – to produce the list of exported symbols from a set of header files:

But without going into further details I can tell you that there are two functions that should be exported that are not, and the dvdinput_ series that shouldn’t have been exposed are. So there are a few things to fix there for sure.

As I said before, my personal preference would be to merge libdvdread and libdvdnav again (they were split a long time ago as some people didn’t need/want the menu support) — if it wasn’t for obvious legal issues I would merge libdvdcss as well, but that’s a different story. I just need to find the motivation to go look in the reverse dependencies of these two libraries, and see if the interface exposed between the two is ever used, it might be possible to reduce their surface as well.

Yes this would be a relatively big change for relatively small gain, on the other hand, it might be worth to get this as a new side-by-side installable library that can be used preferentially, falling back to the old ones if not present. And given the staleness of the code, I wouldn’t really mind having to go through testing from scratch at this point.

Anyway, at least the build system of the three libraries will soon look similar enough that they seem to be part of the same project, instead of each going its own way — among other things the ebuilds for the three should look almost entirely identical, in my opinion, so that should be a good start.

If you want to contribute, given that the only mailing list we have on videolan is for libdvdcss, you can push your branches to Gitorious thanks to the VideoLan mirror and from there just contact somebody in #videolan on Freenode to get it reviewed/merged.

I've now been with the Linux Foundation for just over a year. When I
started, I posted a list of how you can watch to see what I've been
doing. But, given that people like to see year-end-summary
reports, the excellent graphic designers at the Linux Foundation have
put together an image summarizing my past year, in numbers:

In a try to revive the Gentoo Bugday I wrote this article in order to give some guide lines and encourage both users and developers to join. I think it would be great to get this event back and collaborate. Of course everyone can open/close bugs silently but this type of event is a good way to close bugs, attract new developers/users and improve community relations. There is no need to be a Gentoo expert. So I will give you some information about the event.

About:

Bugday is a monthlyonline event that takes place every first Saturday of every month in #gentoo-bugs in the Freenode network. Its goal is to have users and developers collaborate to close/open bugs, update current packages and improve documentation.

Location:

Gentoo Bugday take place in our official IRC channel #gentoo-bugs @ Freenode. You can talk about almost everything. Your ebuilds, version bumps, bugs that you will choose to fix, etc.. This is a 24h event, so don’t worry about the timezone difference.

As one of my four talks at FOSDEM, I gave one on Gentoo titled “Package management and creation in Gentoo Linux.” The basic idea was, what could packagers and developers of other, non-Gentoo distros learn from Gentoo’s packaging format and how it’s iterated on that format multiple times over the years. It’s got some slides but the interesting part is where we run through actual ebuilds to see how they’ve changed as we’ve advanced through EAPIs (Ebuild APIs), starting at 16:39.

If you click through to YouTube, the larger (but not fullscreen) version seems to be the easiest to read.

It was scaled from 720×576 to a 480p video, so if you find it too hard to read the code, you can view the original WebM here.

The Gentoo project has its own official wiki for some time now, and we are going to use it more and more in the next few months. For instance, in the last Gentoo Hardened meeting, we already discussed that most user-oriented documentation should be put on the wiki, and I’ve heard that there are ideas on moving Gentoo project pages at large towards the wiki. And also for the regular Gentoo documentation I will be moving those guides that we cannot maintain ourselves anymore easily towards the wiki.

To support migrations of documents, I created a gxml2wiki.xsl stylesheet. Such a stylesheet can be used, together with tools like xsltproc, to transform GuideXML documents into text output somewhat suitable for the wiki. It isn’t perfect (far from it actually) but at least it allows for a more simple migration of documents with minor editing afterwards.

Currently, using it is as simple as invoking it against the GuideXML document you want to transform:

~$ xsltproc gxml2wiki.xsl /path/to/document.xml

The output shown on the screen can then be used as a page. The following things still need to be corrected manually:

Whitespace is broken, sometimes there are too many newlines. I had to make the decision to put in newlines when needed (which makes too many newlines) rather than a few newlines too few (which makes it more difficult to find where to add in).

Links need to be double/triple checked, but i’ll try to fix that in later editions of the stylesheet

Commands will have “INTERNAL” in them – you’ll need to move the commands themselves into the proper location and only put the necessary output in the pre-tags. This is because the wiki format has more structure than GuideXML in this matter, thus transformations are more difficult to write in this regard.

The stylesheet currently automatically adds in a link towards a Server and security category, but of course you’ll need to change that to the proper category for the document you are converting.

I guess many people may hit similar problems, so here is my experience of the upgrades. Generally it was pretty smooth, but required paying attention to the details and some documentation/forums lookups.

udev-171 -> udev-197 upgrade

Make sure you have CONFIG_DEVTMPFS=y in kernel .config, otherwise the system becomes unbootable for sure (I think the error message during boot mentions that config option, which is good).

The ebuild also asks for CONFIG_BLK_DEV_BSG=y, not sure if that's strictly needed but I'm including it here for completeness.

Things work fine for me without DEVTMPFS_MOUNT. I haven't tried with it enabled, I guess it's optional.

I do not have a split /usr. YMMV then if you do.

Make sure to run "rc-update del udev-postmount".

Expect network device names to change (I guess this is a non-issue for systems with a single network card). This can really mess up things in quite surprising ways. It seems /etc/udev/rules.d/70-persistent-net.rules no longer works (bug #453494). Note that the "new way" to do the same thing (http://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames) is disabled by default in Gentoo (see /etc/udev/rules.d/80-net-name-slot.rules). For now I've adjusted my firewall and other configs, but I think I'll need to figure out the new persistent net naming system.

It can be really non-obvious what to do with this one. Change your rules from e.g. "-m state --state RELATED" to "-m conntrack --ctstate RELATED". See http://forums.gentoo.org/viewtopic-t-940302.html for more info. Also note that iptables-restore doesn't really provide good error messages, e.g. "iptables-restore: line 48 failed". I didn't find a way to make it say what exactly was wrong (the line in question was just a COMMIT line, it didn't actually identify the real offending line). These mysterious errors are usually caused by missing kernel support for some firewall features/targets.

two upgrades together

Actually what adds to the confusion is having these two upgrades done simultaneously. This makes it harder to identify which upgrade is responsible for which breakage. For an even smoother ride, I'd recommend upgrading iptables first, making sure the updated rules work, and then proceed with udev.

We've generated a new set of profiles for Gentoo installation. These are now called 13.0 instead of 10.0, e.g., "default/linux/amd64/10.0/desktop" becomes "default/linux/amd64/13.0/desktop".Everyone should upgrade as soon as possible. This brings (nearly) no user-visible changes. Some new files have been added to the profile directories that make it possible for the developers to do more fine-grained use flag masking (see PMS-5 for the details), and this formally requires a new profile tree with EAPI=5 (and a recent portage version, but anything since sys-apps/portage-2.1.11.31 should work and anything since sys-apps/portage-2.1.11.50 should be perfect).Since the 10.0 profiles will be deprecated immediately and removed in a year, emerge will suggest a replacement on every run. I strongly suggest you just follow that recommendation.One additional change has been added to the package: the "server" profiles will be removed; they do not exist in the 13.0 tree anymore. If you have used a server profile so far, you should migrate to its parent, i.e. from "default/linux/amd64/10.0/server" to "default/linux/amd64/13.0". This may change the default value of some use-flags (the setting in "server" was USE="-perl -python snmp truetype xml"), so you may want to check the setting of these flags after switching profile, but otherwise nothing happens.

While on my machine KDE 4.10.0 runs perfectly fine, unfortunately a lot of Gentoo users see immediate crashes of plasma-desktop - which makes the graphical desktop environment completely unuseable. We know more or less what happened in the meantime, just not how to properly fix it...The problem:

plasma-desktop uses a new code path in 4.10, which triggers a Qt bug leading to immediate SIGSEGV.

The Qt bug only becomes fatal for some compiler options, and only on 64bit systems (amd64).

The Qt bug may be a fundamental architectural problem that needs proper thought.

Reverting the commit to plasma-workspace that introduced the problem makes the crash go away, but plasma-desktop starts hogging 100% CPU after a while. (This is done in plasma-workspace-4.10.0-r1 as a stopgap measure.) Kinda makes sense since the commit was there to fix a problem - now we hit the original problem.

The bug seems not to occur if Qt is compiled with CFLAGS="-Os". Cause unknown.

David E. Narváez aka dmaggot wrote a patch for Qt that fixes this particular codepath but likely does not solve the global problem.

Our Gentoo Qt team understandably only wants to apply a patch if it has been accepted upstream.

Right now, the only option we (as Gentoo KDE team) have is wait for someone to pick up the phone. Either from KDE (to properly use the old codepath or provide some alternative), or from Qt (to fix the bug or apply a workaround)...

Update! Update! Read all about it!You can find the recent updates in a tree near you. They are currently keyworded, but will be stablized as soon as the arch teams find time to do so. You may not want to wait that long as it is a Denial of Service, which is not as severe as it sounds in this case. The user would have to be logged in to cause a DoS.

There have been some other updates to the PostgreSQL ebuilds as well. PostgreSQL will no longer restart if you restarted your system logger. The ebuilds install PAM service files unique to each slot so you don’t have to worry about it being removed when you uninstall an old slot. And, finally, you can write your PL/Python in Python 3.

There's been a lot of information scattered around the internet about
these topic recently, so here's my attempt to put them all in one place
to (hopefully) settle things down and give my inbox a break.

Both of these articles allude to the fact that I'm working on putting
the D-Bus protocol into the kernel, in order to help achieve these
larger goals of proper IPC for applications. And I'd like to confirm
that yes, this is true, but it's not going to be D-Bus like you know it
today.

Our goal (and I use "goal" in a very rough term, I have 8 pages of
scribbled notes describing what we want to try to implement here), is to
provide a reliable multicast and point-to-point messaging system for the
kernel, that will work quickly and securely. On top of this kernel
feature, we will try to provide a "libdbus" interface that allows
existing D-Bus users to work without ever knowing the D-Bus daemon was
replaced on their system.

"But Greg!" some of you will shout, "What about the existing AF_BUS
kernel patches that have been floating around for a while and that you
put into the LTSI 3.4 kernel release?"

The existing AF_BUS patches are great for users who need a very
low-latency, high-speed, D-Bus protocol on their system. This includes
the crazy automotive Linux developers, who try to shove tens of
thousands of D-Bus messages through their system at boot time, all while
using extremely underpowered processors. For this reason, I included
the AF_BUS patches in the LTSI kernel release, as that limited
application can benefit from them.

Please remember the LTSI kernel is just like a distro kernel, it has no
relation to upstream kernel development other than being a consumer of
it. Patches are in this kernel because the LTSI member groups need
them, they aren't always upstream, just like all Linux distro kernels
work.

However, given that the AF_BUS patches have been rejected by the
upstream Linux kernel developers, I advise that anyone relying on them
be very careful about their usage, and be prepared to move away from
them sometime in the future when this new "kernel dbus" code is properly
merged.

As for when this new kernel code will be finished, I can only respond
with the traditional "when it is done" mantra. I can't provide any
deadlines, and at this point in time, don't need any additional help
with it, we have enough people working on it at the moment. It's
available publicly if you really want to see it, but I'll not link to it
as it's nothing you really want to see or watch right now. When it
gets to a usable state, I'll announce it in the usual places
(linux-kernel mailing list) where it will be torn to the usual shreds
and I will rewrite it all again to get it into a mergable state.

In the meantime, if you see me at any of the many Linux conferences I'll
be attending around the world this year, and you are curious about the
current status, buy me a beer and I'll be glad to discuss it in person.

If there's anything else people are wondering about this topic, feel free
to comment on it here on google+, or email me.

It’s been a while again, so time for another Gentoo Hardened online progress meeting.

Toolchain

GCC 4.8 is on development stage 4, so the hardened patches will be worked on next week. Some help on it is needed to test the patches on ARM, PPC and MIPS though. For those interested, keep a close eye on the hardened-dev overlay as those will contain the latest fixes. When GCC 4.9 starts development phase 1, Zorry will again try to upstream the patches.

With the coming fixes, we might probably (need to) remove the various hardenedno* GCC profiles from the hardened Gentoo profiles. This shouldn’t impact too many users as ebuilds add in the correct flags anyhow (for instance when needing to turn off PIE/PIC).

Kernel, grSecurity and PaX

The kernel release 3.7.0 that we have stable in our tree has seen a few setbacks, but no higher version is stable yet (mainly due to the stabilization period needed). 3.7.4-r1 and 3.7.5 are prime candidates with good track record,
so we might be stabilizing 3.7.5 in the very near future (next week probably).

On the PaX flag migration (you know, from ELF-header based marking to extended attributes marking), the documentation has seen its necessary upgrades and the userland utilities have been updated to reflect the use of xattr markings. The eclass we use for the markings will use the correct utility based on the environment.

One issue faced when trying to support both markings is that some actions (like the “paxctl -Cc” which creates the PT_PAX header if it is missing) make no sense with the other (as there is no header when using XATTR_PAX). The eclass will be updated to ignore these flags when XATTR_PAX is selected.

SELinux

Revision 10 is stable in the tree, and revision 11 is waiting stabilization period. A few more changes have been put in the policy repository already (which are installed when using the live ebuilds) and will of course be part of
revision 12.

A change in the userland utilities was also pushed out to allow permissive domains (so run a single domain in permissive mode instead of the entire system).

Finally, the SELinux eclass has been updated to remove SELinux modules from all defined SELinux module stores if the SELinux policy package is removed from the system. Before that, the user had to remove the modules from the store himself manually, but this is error-prone and easily forgotten, especially for the non-default SELinux policy stores.

Profiles

All hardened subprofiles are marked as deprecated now (you’ve seen the discussions on this on the mailinglist probably on this) so we now have a sane set of hardened profiles to manage. The subprofiles were used for things like
“desktop” or “server”, whereas users can easily stack their profiles as they see fit anyhow – so there was little reason for the project to continue managing those subprofiles.

Also, now that Gentoo has released its 13.0 profile, we will need to migrate our profiles to the 13.0 ones as well. So, the idea is to temporarily support 13.0 in a subprofile, test it thoroughly, and then remove the subprofile and switch the main one to 13.0.

System Integrity

The documentation for IMA and EVM is available on the Gentoo Hardened project site. They currently still refer to the IMA and EVM subsystems as development-only, but they are available in the stable kernels now. Especially the default policy that is available in the kernel is pretty useful. When you want to consider custom policies (for instance with SELinux integration) you’ll need a kernel patch that is already upstreamed but not applied to the stable kernels yet.

To support IMA/EVM, a package called ima-evm-utils is available in the hardened-dev overlay, which will be moved to the main tree soon.

Documentation

As mentioned before, the PaX documentation has seen quite a lot of updates. Other documents that have seen updates are the Hardened FAQ, Integrity subproject and SELinux documentation although most of them were small changes.

Another suggestion given is to clean up the Hardened project page; however, there has been some talk within Gentoo to move project pages to the Gentoo wiki. Such a move might make the suggestion easier to handle. And while on the subject of the wiki, we might want to move user guides to the wiki already.

Bugs

Bug 443630 refers to segmentation faults with libvirt when starting Qemu domains on a SELinux-enabled host. Sadly, I’m not able to test libvirt myself so either someone with SELinux and libvirt
expertise can chime in, or we will need to troubleshoot it by bug (using gdb, strace’ing more, …) which might take quite some time and is not user friendly…

Media

Various talks where held at FOSDEM regarding Gentoo Hardened, and a lot of people attended those talks. Also the round table was quite effective, with many users interacting with developers all around. For next year, chances are very high that we’ll give a “What has changed since last year” session and a round table again.

With many thanks to the usual suspects: Zorry, blueness, prometheanfire, lejonet, klondike and the several dozen contributors that are going to kill me for not mentioning their (nick)names.

Preface: It appears that I have fallen behind in my writings. It’s a shame really because I think of things that I should write in the moment and then forget. However, as I’m embracing slowish travel, sometimes I just don’t really do anything that is interesting to write about every day/week.

My last post was about my time in Greece. Since then I have been to Istanbul, Dubai, and (now) Sri Lanka. I was in Istanbul for about 10 days. My lasting impressions of Istanbul were:

+: Istanbul was the first Muslim country I’ve been to. This is is a positive because it opened up some thoughts of what to expect as I continue east. To see all the impressive mosques, to hear the azan (call to prayer) in the streets, to talk to some Turks about religion, really made it a new experience for me.

+: Istanbul receives many visitors per year, which makes it such that it is easy to converse, find stuff you need, etc

-: Istanbul receives many visitors per year, which makes it very touristy in some parts.

+: Istanbul is a huge city and there is much to see. I stepped on Asia for the first time. There are many old, old, buildings that leave you in awe. Oldest shopping area in the world, the Grand Bazaar, stuff like that.

-: Istanbul is a huge city and the public transit is not well connected, I thought.

–: Every shop owner harasses you to come in the store! The best defense that I can recommend is to walk with a purpose (like you are running an errand) but not in a hurry. This will bring the least amount of attention to yourself at risk of “missing” the finer details as you meander.

Let’s not joke anyone, Dubai was a skydiving trip, for sure. I spent 15 days in Dubai and made 30 jumps. It was a blast. I was at the dropzone most everyday and on the weather days, my generous hosts showed me around the city. I didn’t feel the need to take any pictures of the sites because, while impressive, they seemed too “fake” to me (outrageous, silly, etc). I went to the largest mall in the world, ate brunch in the shadow of the largest building in the world, largest aquarium, indoor ski hill in a desert, eventually it was just…meh. However, I will never forget “The Palm”

When deciding where to go onwards, as I knew I shouldn’t stay in Dubai too long (money matters, of course, I would spend my whole lot on fun and there is so much more to see). I ended up in Sri Lanka, because skyscanner told me there was a direct flight there on a budget airline. I don’t see the point in accepting layovers in my flight details at my pace. Then I found someone on HelpX that wanted an English teacher in exchange for accommodation. While I’m not a teacher, I am a native speaker, and that was acceptable at this level of classes. I did a week stint of that in a small village and now I’m relaxing at the beach…I’ll write more about Sri Lanka later and post pics, a fun photo so far:

I had this post in the Drafts for a while, but now it’s time to publish it since the situation does not seem to be improving at all.

As you probably now, if you want to become a Gentoo developer, you need to find yourself a mentor[1]. This used to be easy. I mean, all you had to do was to contact the teams you were interested in contributing as a developer and then one of the team members would step up and help you with your quizzes. However, lately, I find myself in the weird situation of having to become a mentor myself because potential recruits come back to recruiters and say that they could not find someone from the teams to help them. This is sub-optimal for a couple of reasons. First of all, time constrains Mentoring someone can take days, weeks or months. Recruiting someone after being trained (properly or not), can also take days, weeks or months. So somehow, I ended up spending twice as much time as I used to. So we are back to those good old days, where someone needed to wait months before we fully recruit him. Secondly, a mentor and a recruiter should be different persons. This is necessary for recruits to gain a wider and more efficient training as different people will focus on different areas during this training period.

One may wonder, why teams are not willing to spend time to train new developers. I guess, this is because training people takes quite a lot of someone’s time and people tend to prefer fixing bugs and writing code than spending time training people. Another reason could be that teams are short on manpower, so try are mostly busy with other stuff and they just can’t do both at the same time. Others just don’t feel ready to become mentors which is rather weird because every developer was once a mentee. So it’s not like they haven’t done something similar before. Truth is that this seems to be a vicious circle. No manpower to train people -> less people are trained -> Not enough manpower in the teams.

In my opinion, getting more people on board is absolutely crucial for Gentoo. I strongly believe that people must spend time training new people because a) They could offload work to them ;) and b) it’s a bit sad to have quite a few interested and motivated people out there and not spend time to train them properly and get them on board. I sincerely hope this is a temporary situation and things will become better in the future.

ps: I will be in FOSDEM this weekend. If you are there and you would like to discuss about the Gentoo recruitment process or anything else, come and find me ;)

Let me present an informal an unofficial state of Chromium Open Source packages as I see it. Note a possible bias: I'm a Chromium developer (and this post represents my views, not the projects'), and a Gentoo Linux developer (and Chromium package maintenance lead - this is a team effort, and the entire team deserves credit, especially for keeping stable and beta ebuilds up to date).

Arch Linux - ships stable channel, promptly reacts to security updates. NaCl is enabled, following Gentoo closely - I consider that good, and I'm glad people find that code useful. :) 5 use_system_... gyp switches are enabled. A notable thing is that the PKGBUILD is one of the shortest and simplest among Chromium packages - this seems to follow from The Arch Way. There is also chromium-dev on AUR - it is more heavily based on the Gentoo package, and tracks the upstream dev channel. Uses 19 use_system_... gyp switches.

Debian - ancient 6.x version in Squeeze, 22.x in sid at the time of this writing. This is two major milestones behind, and is missing security updates. Not recommended at this moment. :( If you are on Debian, my advice is to use Google Chrome, since official debs should work, and monitor state of the open source Chromium package. You can always return to it when it gets updated.

Fedora - not in official repositories, but Tom "spot" Callaway has an unofficial repo. Note: currently the version in that repo is 23.x, one major version behind on stable. Tom wrote an article in 2009 called Chromium: Why it isn't in Fedora yet as a proper package, so there is definitely an interest to get it packaged for Fedora, which I appreciate. Many of the issues he wrote about are now fixed, and I hope to work on getting the remaining ones fixed. Please stay tuned!

This is not intended to be an exhaustive list. I'm aware of openSUSE packages, there seems to be something happening for Ubuntu, and I've heard of Slackware, Pardus, PCLinuxOS and CentOS packaging. I do not follow these closely enough though to provide a meaningful "review".

Some conclusions: different distros package Chromium differently. Pay attention to the packaging lag: with about 6 weeks upstream release cycle and each major update being a security one, this matters. Support for NativeClient is another point. There are extension and Web Store apps that use it, and when more and more sites start to use it, this will become increasingly important. Then it is interesting why on some distros some bundled libraries are used even though upstream provides an option to use a system library that is known to work on other distros.

Finally, I like how different maintainers look at each other's packages, and how patches and bugs are frequently being sent upstream.

Just a simple announcement for now. It's a bit messy, but should work :D

I have packaged Openstack for Gentoo and it is now in tree, the most complete packaging is probably for Openstack Swift. Nova and some of the others are missing init scripts (being worked on). If you have problems or bugs, report as normal.

When you talk to a friend, she or he knows you are the person in question. But when you do this a friend far away through computers, you can not be sure.
That's why computers have ways to let you know if the person you are talking to is really the right person.

The ways we use today have one problem: We are not sure that they work. It may be that a bad person knows a way to be able to tell you that he is in fact your friend. We do not think that there are such ways for bad persons, but we are not completely sure.

This is why some people try to find ways that are better. Where we can be sure that no bad person is able to tell you that he is your friend. With the known ways today this is not completely possible. But it is possible in parts.

I have looked at those better ways. And I have worked on bringing these better ways to your computer.

openSUSE 12.3 is getting closer and closer and probably one of the last changes I pushed for MySQL was switching the default MySQL implementation. So in openSUSE 12.3 we will have MariaDB as a default.

If you are following what is going on in openSUSE in regards to MySQL, you probably already know, that we started shipping MariaDB together with openSUSE starting with version 11.3 back in 2010. It is now almost three years since we started providing it. There were some little issues on the way to resolve all conflicts and to make everything work nicely together. But I believe we polished everything and smoothed all rough edges. And now everything is working nice and fine, so it’s time to change something, isn’t it? So let’s take a look of the change I made…

MariaDB as default, what does it mean?

First of all, for those who don’t know, MariaDB is MySQL fork – drop-in replacement for MySQL. Still same API, still same protocol, even same utilities. And mostly the same datafiles. So unless you have some deep optimizations depending on your current version, you should see no difference. And what will switch mean?

Actually, switching default doesn’t mean much in openSUSE. Do you remember the time when we set KDE as a default? And we still provide great Gnome experience with Gnome Shell. In openSUSE we believe in freedom of choice so even now, you can install either MySQL or MariaDB quite simply. And if you are interested, you can try testing beta versions of both – we have MySQL 5.6 and MariaDB 10.0 in server:database repo. So where is the change of default?

Actually, the only thing that changed is that everything now links against MariaDB and uses MariaDB libraries – no change from users point of view. And if you will try to update from system that used to have just one package called ‘mysql’, you’ll end up with MariaDB. And it will be default in LAMP pattern. But generally, you can still easily replace MariaDB with MySQL, if you like Oracle Yes, it is hard to make a splash with a default change if you are supporting both sides well…

What happens to MySQL?

Oracles MySQL will not go away! I’ll keep packaging their version and it will be available in the openSUSE. It’s just not going to be a default, but nothing prevents you from installing it. And if you had it in past and you are going to do just a plain upgrade, you’ll stick to it – we are not going to tell you what to use if you know what you want.

Why?

As mentioned before, being default doesn’t have many consequences. So why the switch? Wouldn’t it break stuff? Is that MariDB safe enough? Well, I’m personally using MariaDB since 2010 with few switches to MySQL and back, so it is better tested from my point of view. I originally switched for the kicks of living on the edge, but in the end I found MariaDB boringly stable (even though I run their latest alpha). I never had any serious issue with it. It also has some interesting goodies that it can offer to it’s user over MySQL. Even Wikipedia decided to switch. And our friends at Fedora are considering it too, but AFAIK they don’t have MariaDB yet in their distribution….

Don’t take it as a complain about MySQL guys and girls at Oracle, I know that they are doing a great job that even MariaDB is based on as they do periodical merges to get newest MySQL and they “just” add some more tweaks, engines and stuff.

So, as I like MariaDB and I think it’s time to move, I, as a maintainer of both, proposed to change the default. There were no strong objections, so we are doing it!

Overview

So overall, yes, we are changing default MySQL provider, but you probably wouldn’t even notice

We are currently working on integrating carbon nanotube nanomechanical systems into superconducting radio-frequency electronics. Overall objective is the detection and control of nanomechanical motion towards its quantum limit. In this project, we've got a PhD position with project working title "Gigahertz nanomechanics with carbon nanotubes" available immediately.

You will design and fabricate superconducting on-chip structures suitable as both carbon nanotube contact electrodes and gigahertz circuit elements. In addition, you will build up and use - together with your colleagues - two ultra-low temperature measurement setups to conduct cutting-edge measurements.

Good knowledge of electrodynamics and possibly superconductivity are required. Certainly helpful is low temperature physics, some sort of programming experience, as well as basic familiarity with Linux. The starting salary is 1/2 TV-L E13.

The combination of localized states within carbon nanotubes and superconducting contact materials leads to a manifold of fascinating physical phenomena and is a very active area of current research. An additional bonus is that the carbon nanotube can be suspended, i.e. the quantum dot between the contacts forms a nanomechanical system. In this research field a PhD position is immediately available; the working title of the project is "A carbon nanotube as a moving weak link".

You will develop and fabricate chip structures combining various superconductor contact materials with ultra-clean, as-grown carbon nanotubes. Together with your colleagues, you will optimize material, chip geometry, nanotube growth process, and measurement electronics. Measurements will take place in one of our ultra-low temperature setups.

Good knowledge of superconductivity is required. Certainly helpful is knowledge of semiconductor nanostructures and low temperature physics, as well as basic familiarity with Linux. The starting salary is 1/2 TV-L E13.

Well, all of MythTV 0.26 is now in portage, masked for testing for a few days.

If anyone is interested now is a good time to give it a try and report any issues you find. If all is quiet the masks will come off and we’ll be up-to-date (including all patches up to a few days ago).

Thanks to all who have contributed to the 0.26 bug. I can also happily report that I’m running Gentoo on my mythtv front-end, which should help me with maintaining things. MiniMyth is a great distro, but it has made it difficult to keep the front- and back-ends in sync.

the task was to combine do and re by nils frahm into a new work. i chopped “re” into loops, and rearranged sections by sight and sound for a deliberately loose feel. the resulting piece is entirely unquantized, with percussion generated from the piano/pedal action sounds of “do” set under the “re” arrangement. the perc was performed with an mpd18 midi controller in real time, and then arranged by dragging individual hits with a mouse. since the original piano recordings were improvised, tempo fluctuates at around 70bpm, and i didn’t want to lock myself into anything tighter when creating the downtempo beats.

beats performed live on the mpd18, arranged in ardour3.

normally i’d program everything to a strict grid with renoise, but for this project, i used ardour3 (available in my overlay) almost exclusively, except for a bit of sample preparation in renoise and audacity. the faint background pads/strings were created with paulstretch. my ardour3 session was filled with hundreds of samples, each one placed by hand and nudged around to keep the jazzy feel, as seen in this screenshot:

this is a very rough rework — no FX, detailed mixing/mastering, or complicated tricks. i ran outta time to do all the subtle things i usually do. instead, i spent all my time & effort on the arrangement and vibe. the minimal treatment worked better than everything i’d planned.

It’s not that hard to build a ghc with some exotic target if you have gcc there.

mkGmpDerivedConstants needs to be more cross-compiler friendly It should be really simple to implement, it only queries for data sizes/offsets. I think autotools is already able to do it.

GHC should be able to run llvm with correct -mtriple in crosscompiler case. That way we would get registerised crosscompiler.

Some TODOs:

In order to coexist with native compiler ghc should stop mangling —-target=ia64-unknown-linux-gnu option passed by user and name resulting compiler a ia64-unknown-linux-gnu-ghc and not ia64-unknown-linux-ghc.

That way I could have many flavours of compiler for one target. For example I would like to have x86_64-pc-linux-gnu-ghc as a registerised compiler and x86_64-unknown-linux-gnu-ghc as an unreg one.

I tried not to get into this discussion, mostly because it will degenerate to a mud sliding contest.

Alexis did not take well the fact that Tomáš changed the default provider for libavcodec and related libraries.

Before we start, one point:

I am as biased as Alexis, as we are both involved on the projects themselves. The same goes for Diego, but does not apply to Tomáš, he is just a downstream by transition (libreoffice uses gstreamer that uses *only* Libav).

Now the question at hand: which should be the default? FFmpeg or Libav?

How to decide?

- Libav has a strict review policy every patch goes through a review and has to be polished enough before landing the tree.

- FFmpeg merges daily what had been done in Libav and has a more lax approach on what goes in the tree and how.

- Libav has fate running on most architectures, many of those are running Gentoo, usually real hardware.

- FFmpeg has an old fate with less architectures, many qemu emulations.

- Libav defines the API

- FFmpeg follows adding bits here and there to “diversify”

- Libav has a major release per season, minor releases when needed

- FFmpeg releases a lot touting a lot of *Security*Fixes* (usually old code from the ancient times eventually fixed)

- Libav does care about crashes and fixes them, but does not claim every crash is a Security issue.

- FFmpeg goes by leaps to add MORE features, no matter what (including picking wip branches from my personal github and merging them before they are ready…)

- Libav is more careful, thus having less fringe features and focusing more polishing before landing new stuff.

So if you are a downstream you can pick what you want, but if you want something working everywhere you should target Libav.

If you are missing a feature from Libav that is in FFmpeg, feel free to point me to it and I’ll try my best to get it to you.

Exactly two years ago, a group consisting of the majority of FFmpeg developers took over its maintainership. While I didn’t like the methods, I’m not an insider so my opinion stops here, especially since if you pay attention to who was involved: Luca was part of it. Luca has been a Gentoo developer since probably most of us even used Gentoo and I must admit I’ve never seen him heating any discussion, rather the contrary, and it’s always been a pleasure to work with him. What happened next, after a lot of turmoil, is that the developers split in two groups: libav formed by the “secessionists” and FFmpeg.

Good, so what do we chose now? One of the first things that was done on the libav side was to “cleanup” the API with the 0.7 release, meaning we had to fix almost all its consumers: Bad idea if you want wide adoption of a library that has an history of frequently changing its API and breaking all its consumers. Meanwhile, FFmpeg maintained two branches: the 0.7 branch compatible with the old API and the 0.8 one with the new API. The two branches were supposed to be identical except for the API change. On my side the choice was easy: Thanks but no thanks sir, I’ll stay with FFmpeg.
FFmpeg, while having its own development and improvements, has been doing daily merges of all libav changes, often with an extra pass of review and checks, so I can even benefit from all the improvements from libav while using FFmpeg.

So why should we use libav? I don’t know. Some projects use libav within their release process, so they are likely to be much more tested with libav than FFmpeg. However, until I see real bugs, I consider this as pure supposition and I have yet to see real facts. On the other hand, I can see lots of false claims, usually without any kind of reference: Tomáš claims that there’s no failure that is libav specific, well, some bugstend to saythe contrary and have been open for some time (I’ll get back to XBMC later). Another false claim is that FFmpeg-1.1 will have the same failures as libav-9: Since Diego made a tinderbox run for libav-9, I made the tests for FFmpeg 1.1 and made the failures block our old FFmpeg 0.11 tracker. If you click the links, you will see that the number of blockers is much smaller (something like 2/3) for the FFmpeg tracker. Another false claim I have seen is that there will be libav-only packages: I have yet to see one; the example I had as an answer is gst-plugins-libav, which unfortunately is in the same shape for both implementations.

In theory FFmpeg-1.1 and libav-9 should be on par, but in practice, after almost two years of disjoint development, small differences have started to accumulate. One of them is the audio resampling library: While libswresample has been in FFmpeg since the 0.9 series, libav developers did not want it and made another one, with a very similar API, called libavresample that appeared in libav-9. This smells badly as a NIH syndrome, but to be fair, it’s not the first time such things happen: libav and FFmpeg developers tend to write their own codecs instead of wrapping external libraries and usually achieve better results. The audio resampling library is why XBMCbeing broken with libav is, at least partly, my fault: While cleaning up its API usage of FFmpeg/libav, I made it use the public API for audio resampling, initially with libswresample but made sure it worked with libavresample from libav. At that time, this would mean it required libav git master since libav-9 was not even close to be released, so there was no point in trying to make it compatible with such a moving target. libswresample from FFmpeg was present since the 0.9 series, released more than one year ago. Meanwhile, XBMC-12 has entered its release process, meaning it will probably not work with libav easily. Too late, too bad.

Another important issue I’ve raised is the security holes: Nowadays, we are much more exposed to them. Instead of having to send a specially crafted video to my victim and make him open it with the right program, I only have to embed it in an HTML5 webpage and wait. This is why I am a bit concerned that security issues fixed 7 months ago in FFmpeg have only been fixed with the recently released libav-0.8.5. I’ve been told that these issues are just crashes are have been fixed in a better way in libav: This is probably true but I still consider the delay huge for such an important component of modern systems, and, thanks to FFmpeg merging from libav, the better fix will also land in FFmpeg. I have also been told that this will improve on the libav side, but again, I want to see facts rather than claims.

As a conclusion: Why is the default implementation changed? Some people seem to like it better and use false claims to force their preference. Is it a good idea for our users? Today, I don’t think so (remember: FFmpeg merges from libav and adds its own improvements), maybe later when we’ll have some clear evidence that libav is better (the improvements might be buggy or the merges might lead to subtle bugs). Will I fight to get the default back to FFmpeg ? No. I use it, will continue to use and maintain it, and will support people that want the default back to FFmpeg but that’s about it.

Right at the start the new year 2013 brings the pleasant news that our manuscript "Transversal Magnetic Anisotropy in Nanoscale PdNi-Strips" has found its way into Journal of Applied Physics. The background of this work is - once again - spin injection and spin-dependent transport in carbon nanotubes. (To be more precise, the manuscript resulted from our ongoing SFB 689 project.) Control of the contact magnetization is the first step for all the experiments. Some time ago we picked Pd0.3Ni0.7 as contact material since the palladium generates only a low resistance between nanotube and its leads. The behaviour of the contact strips fabricated from this alloy turned out to be rather complex, though, and this manuscript summarizes our results on their magnetic properties.Three methods are used to obtain data - SQUID magnetization measurements of a large ensemble of lithographically identical strips, anisotropic magnetoresistance measurements of single strips, and magnetic force microscopy of the resulting domain pattern. All measurements are consistent with the rather non-intuitive result that the magnetically easy axis is perpendicular to the geometrically long strip axis. We can explain this by maneto-elastic coupling, i.e., stress imprinted during fabrication of the strips leads to preferential alignment of the magnetic moments orthogonal to the strip direction.

"Transversal Magnetic Anisotropy in Nanoscale PdNi-Strips"D. Steininger, A. K. Hüttel, M. Ziola, M. Kiessling, M. Sperl, G. Bayreuther, and Ch. StrunkJournal of Applied Physics 113, 034303 (2013); arXiv:1208.2163 (PDF[*])[*] Copyright American Institute of Physics. This article may be downloaded for personal use only. Any other use requires prior permission of the author and the American Institute of Physics.

UPDATE: Added some basic migration instructions to the bottom.
UPDATE2: Removed mplayer incompatibility mention. Mplayer-1.1 works with system libav.

As the summary says the default media codec provider for new installs will be libav instead of ffmpeg.

This change is being done due to various reasons like matching default with Fedora and Debian, or due to fact that some projects which are high-profile (eg sh*tload of people use them) will be probably libav only. One example being gst-libav which is in return required by libreoffice-4 which is due release in about month. To go for least pain for the user we decided to move from default ffmpeg to default libav library.

This change won’t affect your current installs at all but we would like to ask you to try to migrate to the libav and test and report any issues. So if stuff happen in the future and we are forced to throw libav as only implementation for everyone you are not left in the dark screaming for your suddenly missing features.

What to do when some package does not build with libav but ffmpeg is fine

There are no such packages left around if I am searching correctly (might be my blindness so do not take my word for it).

So if you encounter any package not building with libav just open bugreport on bugzilla and assign it to media-video team and add lu_zero[at]gentoo.org to CC to be sure he really takes a sneaky look to fix it. If you want to fix the issue yourself it gets even better. You write the patch open the bug in our bugzie and someone will include it. Also the patch should be sent to upstream for inclusion, so we don’t have to keep the patches in tree for long time.

What should I do when I have some issues with libav and I require more features that are on ffmpeg but not on libav

Its easier than fixing bugs about failing packages. Just nag to lu_zero (mail hidden somewhere in this post ;-)) and read this.

So when is this stuff going to ruin my day?

The switch in the tree and news item informing all users of media-video/ffmpeg will be created at the end of the January or early February, unless something really bad happens while you guys test it now.

I feel lucky and I want to switch right away so I can ruin your day by reporting bugs

Great I am really happy you want to contribute. The libav switch is pretty easy to be done as there are only 2 things to keep in mind.

You have to sync your useflags between virtual/ffmpeg and the newly-to-be-switched media-video/libav. This is most probably best to do just edit your package.use stuff and replace the media-video/ffmpeg line with media-video/libav one.

Then one would go straight away for emerge libav but there is one more caveat. Libav has split libpostproc library while ffmpeg still is using the internal one. Code wise they are most probably equal but you have to take account for it so just call emerge with both libraries.emerge -1v libav libpostproc

If this succeeds you have to revdep-rebuild the packages you have or use @preserved-rebuild from portage-2.2 to rebuild all the RDEPENDS of libav.

Many times, when I had to set the make.conf on systems with particular architectures, I had a doubt on which is the best –jobs value.
The handook suggest to have ${core} + 1, but since I’m curious I wanted to test it by myself to be sure this is right.

To make a good test we need a package with a respectable build system that respects the make parallelization and takes at least few minutes to compile. Otherwise with packages that compile in few seconds we are unable to track the effective difference.kde-base/kdelibs is, in my opinion, perfect.

If you are on architecture which kde-base/kdelibs is unavailable, just switch to another cmake-based package.

Now, download best_makeopts from my overlay. Below an explanation on what the script does and various suggestions.

You need to compile the package on a tmpfs filesystem and, I’m assuming you have /tmp mounted as a tmpfs too;

You need to have the tarball of the package on a tmpfs because if you have a slow disk, it may takes more time.

You need to switch your governor to performance.

You need to be sure you don’t have strange EMERGE_DEFAULT_OPTS.

You need to add ‘-B’ because we don’t want to include the time of the installation.

You need to drop the existent cache before compile.

As you can see, the for will emerge the same package with makeopts from 1 to 10. If you have, for example, a single core machine, just try the for from 1 to 4 is enough.

Please, during the test, don’t use the cpu for other purposes, and if you can, stop all services and make the test from the tty; you will see the time for every merge.

I tested this script on ~20 different machines and in the majority of the cases, the best optimization was ${core} or more exactly ${threads} of your CPU.

Conclusion:
From the handbook:

A good choice is the number of CPUs (or CPU cores) in your system plus one, but this guideline isn’t always perfect.

I don’t know who, years ago, suggested in the handbook ${core} + 1 and I don’t want to trigger a flame. I’m just saying, ${core} + 1 is not the best optimization for me and the test confirms the part:“but this guideline isn’t always perfect”

In all cases ${threads} + ${X} is slower than only ${threads}, so don’t use -j20 if you have a dual-core cpu.

Also, I’m not saying to use ${threads}, I’m just saying feel free to make your tests to watch what is the best optimization.

If you have suggestions to improve the functionality of the script or you think that this script is wrong, feel free to comment or leave an email.

Through both LWN and netzpolitik.org I just heard that Aaron Swartz has committed suicide. While watching his speech “How we stopped SOPA” his name ring a bell with me, I looked into my inbox and found that he and I once had a brief chat on html2text, I piece of free software of his that I was in touch with in the context of Gentoo Linux. So there is this software, his website, these past mails, this amazing talk, his political work that I didn’t know about… and he’s dead. It only takes a few minutes of watching the talk to get the feeling that this is a great loss to society.