Qt 5.2.1 landed in Ubuntu 14.04 LTS last Friday, hooray! Making it into a drop-in replacement for Qt 5.0.2 was not trivial. Because of the qreal change, it was decided to rebuild everything against the new Qt, so it was an all at once approach involving roughly 130 source packages while the parts were moving constantly. The landing last week meant pushing to archives around three thousand binary packages - counting all six architectures - with the total size of closer to 10 gigabytes.

The new Qt brings performance and features to base future work on, and is a solid base for the future of Ubuntu. You may be interested in the release notes for Qt 5.2.0 and 5.2.1. The Ubuntu SDK got updated to Qt Creator 3.0.1 + new Ubuntu plugin at the same time, although updates for the older Ubuntu releases is a work in progress by the SDK Team.

How We Got Here

Throughout the last few months before the last joint push, I filed tens of tagged bugs. For most of that time I was interested only in build and unit test results, since even tracking those was quite a task. I offered simple fixes here and there myself, if I found out a fix.

I created automated Launchpad recipe builds for over 80 packages that rely on Qt 5 in Ubuntu. Meanwhile I also kept on updating the Qt packaging for its 20+ source packages and tried to stay on top of Debian's and upstream's changes.

Parallel to this work, some like the Unity 8 and UI Toolkit developers started experimenting with my Qt 5.2 PPA. It turned out the rewritten QML engine in Qt 5.2 - V4 - was not entirely stable when 5.2.0 was released, so they worked together with upstream on fixes. It was only after 5.2.1 release that it could be said that V4 worked well enough for Unity 8. Known issues like these slowed down the start of full-blown testing.

Then everything built, unit tests passed, most integration tests passed and things seemed mostly to work. We had automated autopilot integration testing runs. The apps team tested through all of the app store to find out whether some needed fixes - most were fine without changes. On top of the found autopilot test failures and other app issues, manual testing found a few more bugs

Some critical pieces of softwarelike Sudoku needed small fixing

Finally last Thursday it was decided to push Qt in, with a belief that the remaining issues had fixes in branches or not blockers. It turned out the real deployment of Qt revealed a couple of more problems, and some new issues were raised to be blockers, and not all of the believed fixes were really fixing the bugs. So it was not a complete success. Considering the complexity of the landing, it was an adequate accomplishment however.

Specific Issues

Throughout this exercise I bumped into more obstacles that I can remember, but those included:

Not all of the packages had seen updates for months or for example since last summer, and since I needed to rebuild everything I found out various problems that were not related to Qt 5.2

Unrelated changes during 14.04 development broke packages - like one wouldn't immediately think a gtkdoc update would break a package using Qt

Syncing packaging with Debian is GOOD, and the fixes from Debian were likewise excellent and needed, but some changes there had effects on our wide-spread Qt 5 usage, like the mkspecs directory move

xvfb used to run unit tests needed parameters updated in most packages because of OpenGL changes in Qt

arm64 and ppc64el were late to be added to the landing PPA. Fixing those archs up was quite a last minute effort and needed to continue after landing by the porters. On the plus side, with Qt 5.2's V4 working on those archs unlike Qt 5.0's V8 based Qt Declarative, a majority of Unity 8 dependencies are now already available for 64-bit ARM and PowerPC!

While Qt was being prepared the 100 other packages kept on changing, and I needed to keep on top of all of it, especially during the final landing phase that lasted for two weeks. During it, there was no total control of "locking" packages into Qt 5.2 transition, so for the 20+ manual uploads I simply needed to keep track of whether something changed in the distribution and accommodate.

One issue related to the last one was that some things needed were in progress at the time. There was no support for automated AP test running using a PPA. There was also no support on building images. If migration to Ubuntu Touch landing process (CI Train, a middle point on the way to CI Airlines) had been completed for all the packages earlier, handling the locking would have been clearer, and the "trunk passes all integration tests too" would have prevented "trunk seemingly got broken" situations I ended up since I was using bzr trunks everywhere.

Qt 5.3?

We are near to having a promoted Ubuntu image for the mobile users using Qt 5.2, if no new issues pop up. Ubuntu 14.04 LTS will be released in a month to the joy of desktop and mobile users alike.

It was discussed during the vUDS that Qt 5.3.x would be likely Qt version for the next cycle, to be on the more conservative side this time. It's not entirely wrong to say we should have migrated to Qt 5.1 in the beginning of this cycle and only consider 5.2. With 5.0 in use with known issues, we almost had to switch to 5.2.

Kubuntu will join the Qt 5 users next cycle, so it's no longer only Ubuntu deciding the version of Qt. Hopefully there can be a joint agreement, but in the worst case Ubuntu will need a separate Qt version packaged.
Read more

Background

I upgraded from Linux 3.8 to 3.11 among with newer Mesa, X.Org and Intel driver recently and I found a small workaround was needed because of upstream changes.

The upstream change was the Add "Automatic" mode for "Broadcast RGB" property, and defaulting to the Automatic. This is a sensible default, since many (most?) TVs default to the more limited 16-235, and continuing to default to Full from the driver side would mean wrong colors on the TV. I've set my screen to support the full 0-255 range available to not cut the amount of available shades of colors down.

Unfortunately it seems the Automatic setting does not work for my HDMI input, ie blacks become grey since the driver still outputs the more limited range. Maybe there could be something to improve on the driver side, but I'd guess it's more about my 2008 Sony TV actually having a mode that the standard suggests limited range for. I remember the TV did default to limited range, so maybe the EDID data from TV does not change when setting the RGB range to Full.

I hope the Automatic setting works to offer full range on newer screens and the modes they have, but that's probably up to the manufacturers and standards.

Below is an illustration of the correct setting on my Haswell CPU. When the Broadcast RGB is left to its default Automatic setting, the above image is displayed. When set to Full, the image below with deeper blacks is seen instead. I used manual settings on my camera so it's the same exposure.

Workaround

For me the workaround has evolved to the following so far. Create a /etc/X11/Xsession.d/95fullrgb file:

And since I'm using lightdm, adding the following to /etc/lightdm/lightdm.conf means the flicker only happens once during bootup:

display-setup-script=/etc/X11/Xsession.d/95fullrgb

Important: when using the LightDM setting, enable executable bits (chmod +x) to /etc/X11/Xsession.d/95fullrgb for it to work. Obviously also check your output, for me it was HDMI3.

If there is no situation where it'd set back to "Limited 16:235" setting on its own, the display manager script should be enough and having it in /etc/X11/Xsession.d is redundant and slows login time down. I think for me it maybe went from 2 seconds to 3 seconds since executing xrandr query is not cheap.

Misc

Note that unrelated to Full range usage, the Limited range at the moment behaves incorrectly on Haswell until the patch in bug #71769 is accepted. That means, the blacks are grey in Limited mode even if the screen is also set to Limited.

I'd prefer there would be a kernel parameter for the Broadcast RGB setting, although my Haswell machine does boot so fast I don't get to see too many seconds of wrong colors...
Read more

The update has two fixes. The first one fixes a compiz CPU usage regression. The second one enables unredirection also for Intel and Nouveau users using the Mesa 9.0.x stack. That means up-to-date installs from 12.04.2 LTS installation media and anyone with original 12.04 LTS installation who has opted in to the 'quantal' package updates of the kernel, X.Org and mesa *)

The new default setting for the unredirection blacklist is shown in the image below (CompizConfig Settings Manager -> General -> OpenGL). It now only blacklists the original Mesa 8.0.x series for nouveau and intel, plus the '9.0' (not a point release).

Reading into the results, Mesa 9.0.3 seems to have improved the slowdown in the redirected case. That would include normal desktop usage as well. Meanwhile the unredirected performance remains about 10% higher.

*) Packages linux-generic-lts-quantal xserver-xorg-lts-quantal libgl1-mesa-dri-lts-quantal libegl1-mesa-drivers-lts-quantal. 'raring' stack with Mesa 9.1 and kernel 3.8 will be available around the time of 12.04.3 LTS installation media late August.

Oh boy. June stormed in and the May installment is late! Not much changed at the top. The Northern Hemisphere spring storms keep Stormcloud at the top with Fluendo DVD staying put at the number two spot. Steam continues its top of the chart spree on the Free Top 10.

Want to develop for the new Phone and Tablet OS, Ubuntu Touch? Be sure to check out the “Go Mobile” site for details.

The top 10 free apps list contains gratis applications that are distributed under different types of licenses, some of which may not be open source. For detailed license information, please check each application’s description in the Ubuntu Software Center.

Packages

I quite like the current status of Qt 5 in Debian and Ubuntu (the links are to the qtbase packages, there are ca. 15 other modules as well). Despite Qt 5 being bleeding edge and Ubuntu having had the need to use it before even the first stable release came out in December, the co-operation with Debian has gone well. Debian is now having the first Qt 5 uploads done to experimental and later on to unstable. My work contributed to pkg-kde git on the modules has been welcomed, and even though more work has been done there by others, there haven't been drastic changes that would cause too big transition problems on the Ubuntu side. It has of course helped to ask others what they want, like the whole usage of qtchooser. Now with Qt 5.0.2 I've been able to mostly re-sync all newer changes / fixes to my packaging from Debian to Ubuntu and vice versa.

There will remain some delta, as pkg-kde plans to ask for a complete transition to qtchooser so that all Qt using packages would declare the Qt version either by QT_SELECT environment variable (preferable) or a package dependency (qt5-default or qt4-default). As a temporary change related to that, Debian will have a debhelper modification that defaults QT_SELECT to qt4 for the duration of the transition. Meanwhile, Ubuntu already shipped the 13.04 release with Qt 5, and a shortcut was taken there instead to prevent any Qt 4 package breakage. However, after the transition period in Debian is over, that small delta can again be removed.

I will also need to continue pushing any useful packaging I do to Debian. I pushed qtimageformats and qtdoc last week, but I know I'm still behind with some "possibly interesting" git snapshot modules like qtsensors and qtpim.

Patches

More delta exists in the form of multiple patches related to the recent Ubuntu Touch efforts. I do not think they are of immediate interest to Debian – let's start packaging Qt 5 apps to Debian first. However, about all of those patches have already been upstreamed to be part of Qt 5.1 or Qt 5.2, or will be later on. Some already were for 5.0.2.

A couple of months ago Ubuntu did have some patches hanging around with no clear author information. This was a result of the heated preparation for the Ubuntu Touch launches, and the fact that patches flew (too) quickly in place into various PPA:s. I started hunting down the authors, and the situation turned out to be better than I thought. About half of the patches were already upstreamed, and work on properly upstreaming the other ones was swiftly started after my initial contact. Proper DEP3 fields do help understanding the overall situation. There are now 10 Canonical individuals in the upstream group of contributors, and in the last week's sprint it turned out more people will be joining them to upstream their future patches.

Nowadays about all the requests I get for including patches from developers are stuff that was already upstreamed, like the XEmbed support in qtbase. This is how it should be.

One big patch still being Ubuntu only is the Unity appmenu support. There was a temporary solution for 13.04 that forward-ported the Qt 4 way of doing it. This will be however removed from the first 13.10 ('saucy') upload, as it's not upstreamable (the old way of supporting Unity appmenus was deliberately dropped from Qt 5). A re-implementation via QPA plugin support is on its way, but it may be that the development version users will be without appmenu support for some duration. Another big patch is related to qtwebkit's device pixel ratio, which will need to be fixed. Apart from these two areas of work that need to be followed through, patches situation is quite nice as mentioned.

Conclusion

Free software will do world domination, and I'm happy to be part of it.
Read more

I'd like to modify my discussion comment and earlier thoughts into a short blog post touching only some of the technical concerns voiced, and my opinion to those.

Claim (my version): Ubuntu/Canonical is going the "Google route" to become another Android, while Android has not benefited the Linux ecosystem in any way, forking everything

Firstly, Ubuntu is open to development and community for also mobile and tablet - Android has none of that, just code drops that get modded. (yes, some people have a problem with CLA like Canonical's or Qt's, I have no problem with those - let's keep that discussion elsewhere). Ubuntu contributes back to Debian and upstream projects like Qt - those upstream projects it's not upstream of itself. There are not too many free software mobile UIs for example. SHR has some E17 apps, Nemo Mobile a handful of Qt apps and so on.

Secondly, I disagree about Android - even in its current shape and after creating everything from scratch with mobile on mind, Android has done tremendous things for the free software community, kernel development, mobile device driver and making things like Replicant possible. If those aren't directly seen on the desktop side, that's because it's not the desktop and most free software desktop users don't use free software mobile products (usually at most a vendor provided Android).

I feel people get too attached to software projects or even the desktop in general. The money to pay desktop has traditionally largely come from the server. As a discussion-heating example Wayland has been a great promise for 5 years and continues to be, yet no products use it (software products like distributions or hardware+software products). That's not a problem per se for a great and ambitious project, but it means no interested party has taken it to create products. I was very excited about Gallium3D and Wayland in 2008, but somewhat optimistic in believing they would conquer the world in one or two years. In perspective, I've always seen the "version staring" a common habit in enthusiasts me included. I think it extents to "shiny development projects that should be taken into production use immediately".

The Nokia N9 triumphs all other 2011 mobile phones in general and even the current user interfaces like iOS, Android and Windows Phone in general usability ideas (if only it'd run Cortex-A15 instead of OMAP3..). It uses X.org and Qt 4.7. Jolla's plans for their first phone at the end of this year? Qt 4.8, no Wayland. Like N9 which otherwise had unfortunate fate, I hope Jolla will sell millions of free software wielding products to the masses. The biggest problem with X.org is, though, the drivers, generally zero support from vendors so hard to make products. Hooking into Android EGL drivers and building on top of that seems a good compromise at the moment. Note that from product creation point of view it's not the non-shininess of X.org that IMHO is the blocker. Wayland and Mir may help on the driver side.

I'd love to see more push to have actual products on the market, since otherwise we don't get free software to the masses. If Mir helps Ubuntu to do that in one year, fine (I don't know how it's going to be). Yes Mir is a new shiny project, but it's a very product/target oriented project one. If Android would be open as a project, it wouldn't hurt - other than feelings attached to the other projects especially by the core developers and fans of those - if it was the superior alternative from product creation perspective making all of X.org, upstart, systemd, Wayland, Pulseaudio, D-Bus, glibc less interesting to product creators while even more interest would go to Android. It's not so now, Android is not an open project in any sense, even though still beneficial for free software. Ubuntu will keep using a lot more of the traditional stack anyway than Android (which also just got rid of BlueZ), but I have zero problem of changing any of the components if it's visioned to be required to get finished, ready to use products out. IMHO the key is to get products out, and I hope all the parties manage to do that.

Of the traditional GNU/Linux desktop distributions only Ubuntu seems to be adapting for the mobile in large steps at the moment. The other distributions in the mobile playing field are: (Android/)Replicant, Mer/Sailfish, Firefox OS, Tizen, added with OpenEmbedded based distributions like SHR. Have you used those on a daily basis on your devices? I believe you should. I think KDE will bring with its Plasma Active - currently focusing on building on top of Mer - mobile power to the traditional GNU/Linux distributions, but otherwise it's all up to the new players - and Ubuntu.

Like many know, I used Debian exclusively on my primary phone for ca. two years before switching mostly to N9. During all that time, I already pondered why people and distributions are so focused on x86 and desktop. And the reason is that that's what their history is, and I stared at the wrong place - desktop distributions. I dismissed Android and some of the small newcomers in the mobile distro playing field, but it seems that big changes are needed to not need completely new players. I think Ubuntu is on the completely right track to both benefit from the history and adapt for the future. I still hope more developers to Debian Mobile, though!! Debian should be the universal operating system after all.

Disclaimer: I'm an Ubuntu community person from 2004, Debian Developer since 2008 and a contractor for Canonical for ca. 1 year. My opinions haven't changed during the 1 year, but I've learned a lot more of how free software is loved at Canonical despite critics.

Update January 2013: Both Ubuntu 12.04 LTS and Ubuntu 12.10 now have this new feature enabled by default!

Here's an update to my previous entry. In summary, the Compiz update for Ubuntu 12.10 is now in the quantal-proposed updates, and enables unredirection by default for fullscreen applications like games. Happy gaming holidays! A new Compiz update 0.9.7.12 enabling unredirection by default for Ubuntu 12.04 LTS users is in the SRU PPA.

Several changes have happened since the last update, addressing some potential issues uncovered by the people testing the updates (thanks to all!). Daniel has again done all the hard work with regards to actual development.

Changes affecting both 12.10 & 12.04 LTS:

Some drivers do not offer tear-free Xv output without a compositor (or glXSwapBuffers in general). Therefore there is a new option available, for which the default setting enables redirection exception in case of some common video players (eg. Adobe Flash plugin and Totem). The option is Composite -> Undirect Match (unredirect_match) available in the CompizConfig Settings Manager (ccsm). The most notable driver is intel on Intel SB/IVB. Those newer chips don't support the XvPreferOverlay xorg.conf option that would bring tear-free non-composited Xv video on earlier Intels.

The default setting for unredirect_match should be fine for existing users. But if you want to enable unredirection also for the common/default video players and risk tearing on some drivers and some applications, change the unredirect_match option to be just '(any)'. This might help with video playback on older/slower integrated graphics which are only barely powerful enough to play HD videos.

12.04 LTS only:

0.9.7.10 will not be uploaded to precise, it was just used for early testing. 0.9.7.12 currently tested in the SRU PPA enables unredirection by default also on the 12.04 LTS release.

Intel and nouveau mesa drivers are now blacklisted from unredirection on Mesa 8.0 because of so far unresolved driver problems. The blacklist is however configurable in OpenGL -> unredirect_driver_blacklist. The default blacklist regexp is '(nouveau|Intel).*Mesa 8.0'. This does not affect other drivers or Mesa 9.0, so those intel and nouveau users that install 12.04.2 LTS after the end of January or opt-in into the 'LTS-Q' hardware enablement stack for existing installations around the same time will not be blacklisted anymore. Update January 2013: on 12.04 LTS (only), as a last minute change, also Mesa 9.0 in combination with Intel or Nouveau is blacklisted by default at least until LTS-Q stack is properly testable. You can modify the unredirect_driver_blacklist in CCSM -> OpenGL. It was found out that at least mesa 9.0 _only_ from x-updates is not enough - Intel has graphics display problems. It is probable that the problems go away with the full 12.04.2 stack (kernel, X.org, libdrm, mesa) at which point the Intel/nouveau blacklisting can be removed.

The original unredirect_fullscreen_windows option is actually forced on now, because of potential gconf setting migration problems. Disabling unredirection can be done via the unredirect_match option above, by simply blanking the string in there, including removing the '(any)' part - everything will be redirected in that case.

Unity, which uses Compiz and Nux for drawing, recently had a regression with full screen gaming speeds (LP: #1024304). While the performance of Compiz itself was improved significantly in Ubuntu 12.10, the big changes like full OpenGL ES support brought in some regressions at least from benchmarking point of view. Unity/Compiz has had a small performance impact also in Ubuntu 12.04 LTS depending on what it's being compared to. Any compositing method will necessarily bring some performance hit, but there's another option...

Many people know already about enabling the setting "Unredirect Fullscreen Windows" in Compiz's Composite plugin, but having to enable it manually meant that most people didn't get to use it. The feature detects when an application is running full screen, and simply takes compositing out of the equation, improving performance. Getting the feature work fluently has required fixes in both Compiz and drivers, which is why it hasn't been enabled by default before (LP: #1063690).

But things are progressing now. The Unity team's SRU PPA now has a test build of the new Compiz 0.9.8.6 for Ubuntu 12.10, which enables this feature by default:

ppa:unity-team/sruSince Phoronix is a frequent reporter on Linux gaming performance, I (very) quickly run phoronix-test-suite's Open Arena test on my Sandy Bridge machine with the stock Ubuntu 12.10 (quantal) Compiz and with Compiz upgraded from the PPA: 18% increase in fps! (test1_1 has older compiz, test1_2 the newer, no settings touched)

The idea is to get the new Compiz into quantal-proposed after the previous snapshot release 1:0.9.8.4+bzr3412-0ubuntu0.1 gets its bug fixes properly verified via the Stable Release Updates process. Note that also this previous snapshot contains all the needed fixes for unredirecting fullscreen windows, the final tagged 0.9.8.6 version just switches the default to be enabled.

For Ubuntu 12.04 LTS (precise) there will be two steps. First of all, Daniel van Vugt has just backported the required Compiz fixes to the 0.9.7 branch of Compiz that the 12.04 LTS uses and tagged the 0.9.7.10 release. Also in the case of precise, there is an earlier snapshot release in the SRU system, but that one does not yet include the needed backports even though it includes many other fixes. The 0.9.7.10 will be available in the same SRU PPA soon. Secondly, once the fixes are in but the feature is not yet enabled by default, the X.org driver team will need to look at additional fixes for the drivers before enabling the Unredirect Fullscreen Windows by default. But after that, Ubuntu 12.04 LTS should also see a Compiz release with this feature turned on by default.

The repositories here contain sources and binaries for the arm64 bootstrap in Debian (unstable) and Ubuntu (quantal). There are both toolchain and tools packages for amd64 build machines and arm64 binaries built with them. And corresponding sources.

I believe I'm not the only one who thinks that use case oriented Grub2 documentation is hard to find, and a lot of the documentation is obsolete or wrong. My main cause for writing this blog post is a currently unanswered question regarding 2.00, but meanwhile it seems months have passed and still most 1.99 documentation is wrong as well, which might be interesting to some.

The aim is to prevent grub entries from being edited, while not restricting actual booting. This protection is meant for computers not having any confidential stuff, but just wanting to do some light weight security with the assumption that the computer isn't physically opened.

Common setup

You will obviously want to disable any automatically generated root access giving entries, by for example uncommenting GRUB_DISABLE_RECOVERY="true" in /etc/default/grub on Debian or Ubuntu. Also you would disable allowing any external boot devices to be used in BIOS/EFI/coreboot, which you would also have protected with a password. And that often means you need to also disable USB legacy support, since some BIOSes tend to offer all USB devices as bootable without password otherwise (note that I guess that could also cause problems accessing setup on desktop computers if your only keyboard is USB).

1.99

So to first fix the false instructions in variousplaces - no, setting the superuser in 00_header as instructed is not enough. It might be, but does not apply if eg. old kernels are put into submenu (Ubuntu bug 718670, Fedora bug 836259). The protection from editing does not apply there. And if you remove all but one kernel so that there is no submenu, a submenu will be automatically created when there is a new kernel installed via security updates. I didn't need the submenu feature anyway, so I used to comment out the following lines in /etc/grub.d/10_linux:

I hope that was useful. I can imagine it causing a couple of family battles if the commonly instructed setup was the only protection used and there's for example a case of two computer savvy siblings that are eager to get to each others' computers...

2.00 & The Question

The problem with 2.00 is that the superusers setup yields a non-bootable system, ie. password is required for booting. But Google wasn't smiling at me today! Terrible. Can you help me (and others) with 2.00? The aim would be to have a 1.99-like setup where superuser password protects all entries from editing, but booting is fine without any passwords.

Update: Thanks, problem solved, see comments! Find the following line in /etc/grub.d/10_linux: