02 Aug 2015

Code

We fixed a regression in the wrapping layout of side-by-side diffs on bazaar.launchpad.net (#1436483)

Various code pages now have meta tags to redirect go get to the appropriate Bazaar or Git URL, allowing the removal of special-casing from the go tool (#1465467)

Merge proposal diffs including mention of binary patches no longer crash the new-and-improved code review comment mail logic (#1471426), and we fixed some line-counting bugs in that logic as well (#1472045)

Links to the Git code browsing interface now use shorter URL forms

We've also made a fair amount of progress on adding support for triggering webhooks from Launchpad (#342729), which will initially be hooked up for pushes to Git repositories. The basic code model, webservice API, and job retry logic are all in place now, but we need to sort out a few more things including web UI and locking down the proxy configuration before we make it available for general use. We'll post a dedicated article about this once the feature becomes available.

Notifications about duplicate bugs now include an X-Launchpad-Bug-Duplicate header (#363995)

Package build failure notifications now include a "You are receiving this email because …" rationale (#410893)

Package build infrastructure

The sbuild upgrade last month introduced some regressions in our handling of package builds that need to wait for dependencies (e.g. #1468755), and it's taken a few goes to get this right; this is somewhat improved now, and the next builder deployment will fix all the currently-known bugs in this area

In the same area, we've made some progress on adding minimal support for Debian's new build profiles syntax, applying fixes to upload processing and dependency-wait analysis, although this should still be considered bleeding-edge and unlikely to work from end to end

We've been working on adding support for building snap packages (#1476405), but there's still more to do here; we should be able to make this available to some alpha testers around mid-August

Miscellaneous

We've arranged to redirect translations for the overlay PPA used for current Ubuntu phone images to the ubuntu-rtm/15.04 series so that they can be translated effectively (#1463723); we're still working on copying translations into place from before this fix

Projects and project groups no longer have separately-editable "display name" and "title" fields, which were very similar in purpose; they now just have display names (#1853, #4449)

Cancelled live file system builds are sorted to the end of the build history, rather than the start (#1424672)

I was initially annoyed to see implications earlier on Planet Ubuntu that Ubuntu community was in decline. I was tempted to name this article "Why the Negativity? Let's Get On With Making Ubuntu Awesome"

Ubuntu community is not in decline, if you take a broader view and stick to basics. Some (may) continue to focus on a very narrow segment of society (developers mostly) and that's a shame. It's also not the ubuntu I joined. I seem to recall that "We're all one." We do not count certain types of people over others and we should not proclaim the decline of a community when a thin demographic is not increasing in numbers.
Let's define some terms:

VancouverA metropolitan area (city) in British Columbia, Canada.

LocalAn area that is traversable on foot or bike or public transit within 45 minutes.

CommunityA group of people that share an affinity to one another, historically by virtue of being local.

First off, let me say: almost all URLs that were previously working should still work. Only the feed URLs are broken, and this is not something I can fix. If you were following my blog via a feed reader, you should update to the new feed. Sorry for the inconvenience.

Having said that, I'd like to share with you the motivation that made me move and the details of the migration.

The bad things of WordPress

Now, this doesn't want to be a rant, so I'll be pretty concise. WordPress, the content management system, is an excellent platform for blogging. Easy to start with, easy to maintain, easy to use. WordPress.com makes things even easier. It also comes with many useful features, like comments and social networks integration.

The problem is: you can't customize things or add features without paying. Of course, this is business, and I do not want to discuss business decisions made at WordPress.com. Not only that, but I could live fine with most of the major limitations. Also, I was perfectly conscious of this kind of problems with WordPress.com when I started (after all, this is not the first blog I started).

I actually become upset of WordPress.com when writing the series of blog posts about Elliptic Curve Cryptography. When writing these articles, I spent a lot of time employing workarounds to overcome WordPress.com limitations. Being used to Vim and its advanced features, I also found the editors (both the old and the new one) as a great obstacle for getting things done quickly. I do not want to enter the details of the problems I'm referring to, what matters is that, eventually, I gave up and I realized it was time to move on and seek for an alternative.

Why Pelican

Pelican is a static site generator. I've always thought that a static site had too many limitations for me. But while seeking an alternative to WordPress.com, I realized that many of those limitations were not affecting me in any way. Actually, with a static site I can do everything I want: edit my articles with Vim, render my equations with MathJax, customize my theme, version control my content, write scripts to post process my content.

The only bad thing about Pelican is that it does not come with any theme I truly like. I decided to make my own. I'm not entirely satisfied with it, as I feel it is too "anonymous", but I believe it is fully responsive, fast, readable and offers all the features I want. Perhaps I'll tweak it a little more to make it more "personal".

Setting up Pelican and migrating everything required some time, but at least this time I worked on true solutions, not on ugly hacks and workarounds like I did with WordPress. This implies that when writing articles I will be able to focus more on content than other details.

Why not other static site generators

In short: Pelican is written in Python and to my eyes it looked better than the other Python static site generators. I'll be honest and say that I did not truly evaluate all of the alternatives: I knew list.org switched to Pelican and that made me try Pelican before all other solutions.

Conclusion

In the end I decided to leave WordPress for Pelican hosted on GitHub Pages. I'm pretty satisfied with the result I got. The nature of GitHub Pages prevents me from using HTTP redirects (and therefore the old feed links are broken), however in exchange I've got much more freedom, and this is what matters to me.

In my social circle, I'm nearly the only person I know who grew up in area. None of the newcomers I know had heard of hydroplane racing before moving to Seattle. Even after I explain it to them - i.e., boats with 3,000+ horse power airplane engines that fly just above the water at more than 320kph (200mph) leaving 10m+ (30ft) wakes behind them! - most people seem more puzzled than interested.

I grew up near the shore of Lake Washington and could see (and hear!) the races from my house. I don't follow hydroplane racing throughout the year but I do enjoy watching the races at Seafair. Here's my attempt to explain and make the case for the races to new Seattleites.

The wooden U-60 Miss Thriftway circa 1955 (Thriftway is a Washinton-based supermarket that nobody outside has heard of) below is a picture of old-Seattle awesomeness. Modern hydroplanes are now made of fiberglass but two out of three isn't bad.

Although the boats are racing this year in events in Indiana, San Diego, and Detroit in addition to the two races in Washington, hydroplane racing retains deep ties to the region. Most of the drivers are from the Seattle area. Many or most of the teams and boats are based in Washington throughout the year. Many of the sponsors are unknown outside of the state. This parochialness itself cultivates a certain kind of appeal among locals.

In addition to old-Seattle/new-Seattle cultural divide, there's a class divide that I think is also worth challenging. Although the demographics of hydro-racing fans is surprisingly broad, it can seem like Formula One or NASCAR on the water. It seems safe to suggest that many of the demographic groups moving to Seattle for jobs in the tech industry are not big into motorsports. Although I'm no follower of motorsports in general, I've written before cultivated disinterest in professional sports, and it remains something that I believe is worth taking on.

It's not all great. In particular, the close relationship between Seafair and the military makes me very uneasy. That said, even with the military-heavy airshow, I enjoy the way that Seafair weekend provides a little pocket of old-Seattle that remains effectively unchanged from when I was a kid. I'd encourage others to enjoy it as well!

01 Aug 2015

tl;dr: Your Ubuntu-based container is not a copyright violation. Nothing to see here. Carry on.

I am speaking for my employer, Canonical, when I say you are not violating our policies if you use Ubuntu with Docker in sensible, secure ways. Some have claimed otherwise, but that's simply sensationalist and untrue.

Canonical publishes Ubuntu images for Docker specifically so that they will be useful to people. You are encouraged to use them! We see no conflict between our policies and the common sense use of Docker.

Going further, we distribute Ubuntu in many different signed formats -- ISOs, root tarballs, VMDKs, AMIs, IMGs, Docker images, among others. We take great pride in this work, and provide them to the world at large, on ubuntu.com, in public clouds like AWS, GCE, and Azure, as well as in OpenStack and on DockerHub. These images, and their signatures, are mirrored by hundreds of organizations all around the world. We would not publish Ubuntu in the DockerHub if we didn't hope it would be useful to people using the DockerHub. We're delighted for you to use them in your public clouds, private clouds, and bare metal deployments.

Any Docker user will recognize these, as the majority of all Dockerfiles start with these two words....

We explicitly encourage distribution and redistribution of Ubuntu images and packages! We also embrace a very wide range of community remixes and modifications. We go further than any other commercially supported Linux vendor to support developers and community members scratching their itches. There are dozens of such derivatives and many more commercial initiatives based on Ubuntu - we are definitely not trying to create friction for people who want to get stuff done with Ubuntu.

Our policy exists to ensure that when you receive something that claims to be Ubuntu, you can trust that it will work to the same standard, regardless of where you got it from. And people everywhere tell us they appreciate that - when they get Ubuntu on a cloud or as a VM, it works, and they can trust it. That concept is actually hundreds of years old, and we'll talk more about that in a minute....

So, what do I mean by "sensible use" of Docker? In short - secure use of Docker. If you are using a Docker container then you are effectively giving the producer of that container 'root' on your host. We can safely assume that people sharing an Ubuntu docker based container know and trust one another, and their use of Ubuntu is explicitly covered as personal use in our policy. If you trust someone to give you a Docker container and have root on your system, then you can handle the risk that they inadvertently or deliberately compromise the integrity or reliability of your system.

Our policy distinguishes between personal use, which we can generalise to any group of collaborators who share root passwords, and third party redistribution, which is what people do when they exchange OS images with strangers.

Third party redistribution is more complicated because, when things go wrong, there's a real question as to who is responsible for it. Here's a real example: a school district buys laptops for all their students with free software. A local supplier takes their preferred Linux distribution and modifies parts of it (like the kernel) to work on their hardware, and sells them all the PCs. A month later, a distro kernel update breaks all the school laptops. In this case, the Linux distro who was not involved gets all the bad headlines, and the free software advocates who promoted the whole idea end up with egg on their faces.

We've seen such cases in real hardware, and in public clouds and other, similar environments. Digital Ocean very famously published some modified and very broken Ubuntu images, outside of Canonical's policies. That's inherently wrong, and easily avoidable.

So we simply say, if you're going to redistribute Ubuntu to third parties who are trusting both you and Ubuntu to get it right, come and talk to Canonical and we'll work out how to ensure everybody gets what they want and need.

In doing so, that bottle should earn your confidence that it was produced according to strict quality, format, and geographic standards.

Before you pop the cork, check the seal, to ensure it hasn't been opened or tampered with. Now, drink it however you like.

Pour that Champagne over orange juice (if you must). Toss a couple ice cubes in your Scotch (if that's really how you like it). Pour that Bourbon over a Coke (if that's what you want).

Enjoy however you like -- straight up or mixed to taste -- with your own guests in the privacy of your home. Just please don't pour those concoctions back into the bottle, shove a cork in, put them back on the shelf at your local liquor store and try to pass them off as Champagne/Scotch/Bourbon.

Rather, if that's really what you want to do -- distribute a modified version of Ubuntu -- simply contact us and ask us first (thanks for sharing that link, mjg59). We havesomeamazingtools that can help you either avoid that situation entirely, or at least let's do everyone a service and let us help you do it well.

Believe it or not, we're really quite reasonable people! Canonical has a lengthy, public track record, donating infrastructure and resources to many derivative Ubuntu distributions. Moreover, we've successfully contracted mutually beneficial distribution agreements with numerous organizations and enterprises. The result is happy users and happy companies.

My monthly report covers a large part of what I have been doing in the free software world. I write it for my donators (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it's one of the best ways to find volunteers to work with me on projects that matter to me.

Debian LTS

This month I have been paid to work 15 hours on Debian LTS. In that time I did the following:

Finished the work on tracker.debian.org to make it display detailed security status on each supported release (example).

Prepared and released DLA-261-2 fixing a regression in the aptdaemon security update (happening only when you have python 2.5 installed).

Prepared and released DLA-286-1 fixing 1 CVE in squid3. The patch was rather hard to backport. Thankfully upstream was very helpful, he reviewed and tested my patch.

Did one week of "LTS Frontdesk" with CVE triaging. I pushed 19 commits to the security tracker.

Kali Linux / Debian Stretch work

Kali Linux wants to experiment something close to Debian Constantly Usable Testing: we have a kali-rolling release that is based on Debian Testing and we want to take a new snapshot every 4 months (in order to have 3 releases per year).

More specifically we have a kali-dev repository which is exactly Debian Stretch + our own Kali packages (the kali package take precedence) updated 4 times a day, just like testing is. And we have a britney2 setup that generates kali-rolling out of kali-dev (without any requirement in terms of delay/RC bugs, it just ensures that dependencies are not broken), also 4 times a day.

We have jenkins job that ensures that our metapackages are installable in kali-dev (and kali-rolling) and that we can build our ISO images. When things break, I have to fix them and I try to fix them on the Debian side first. So here are some examples of stuff I did in response to various failures:

Reported #791588 on texinfo. It was missing a versioned dependency on tex-common and migrated too early. The package was uninstallable in testing for a few days.

Reported #791591 on pinba-engine-mysql-5.5: package was uninstallable (had to be rebuilt). It appeared on output files of our britney instance.

Reported #791647: debtags no longer supports "debtags update -local" (a feature that went away but that is used by Kali).

I made a NMU of debtags to fix a release critical bug (#791561 debtags: Missing dependency on python3-apt and python3-debian). kali-debtags was uninstallable because it calls debtags in its postinst.

Reported #791874 on python-guess-language: Please add a python 2 library package. We have that package in Kali and when I tried to sync it from Debian I broke something else in Kali which depends on the Python 2 version of the package.

I made a NMU of tcpick to fix a build failure with GCC5 so that the package could go back to testing (it's part of our metapackages).

I requested a bin-NMU of jemalloc and a give-back of hiredis on powerpc in #792246 to fix #788591 (hiredis build failure on powerpc). I also downgraded the severity of #784768 to important so that the package could go back to testing. Hiredis is a dependency of OpenVAS and we need the package in testing.

If you analyze this list, you will see that a large part of the issues we had come down to package getting removed from testing due to RC bugs. We should be able to anticipate those issues and monitor the packages that have an impact on Kali. We will probably add new jenkins job that installs all the metapackages and then run how-can-i-help -s testing-autorm --old… I just submitted #794238 as a wishlist against how-can-i-help.

At the same time, there are bugs that make it into testing and that I fix / work around on the Kali side. But those fixes / work around might be more useful if they were pushed to testing via testing-proposed-updates. I tried to see whether other derivatives had similar needs to see if derivatives could join their efforts at this level but it does not look like so for now.

Last but not least, bugs reported on the Kali side also resulted in Debian improvements:

I reported #793360 on apt: APT::Never-MarkAuto-Sections not working as advertised. And I submitted a patch.

We wanted a newer version of the nvidia drivers. I filed #793079 requesting the new upstream release and the maintainer quickly uploaded it to experimental. I imported it on the Kali side but discovered that it was not working on i386 so I submitted #793160 with a patch.

I noticed that Kali build daemons tend to accumulate many /dev/shm mounts and tracked this down to schroot. I reported it as #793081.

Other Debian work

Sponsorship. I sponsored multiple packages for Daniel Stender who is packaging prospector, a software that I requested earlier (through RFP bug). So I reviewed and uploaded python-requirements-detector, python-setoptconf, pylint-celery and pylint-common. During a review I also discovered a nice bug in dh-python (#793609a comment in the middle of a Build-Depends could break a package). I also sponsored an upload of notmuch-addrlookup (new package requested by a Freexian customer).

Packaging. I uploaded python-django 1.7.9 in unstable and 1.8.3 in experimental to fix security issues. I uploaded a new upstream release of ditaa through a non-maintainer uploaded (again at the request of a Freexian customer).

Distro Tracker. Beside the work to integrate detailed security status, I fixed the code to be compatible with Django 1.8 and modified the tox configuration to ensure that the test suite is regularly run against Django 1.8. I also merged multiple patches of Christophe Siraut (cf #784151 and #754413).

It took a while, but the Unreliable Town Clock finally lived up to its name. Surprisingly, the fault was not mine, but Amazon's.

For several hours tonight, a number of AWS services in us-east-1, including SNS, experienced elevated error rates according to the AWS status page.

Successful, timely chimes were broadcast through the Unreliable Town Clock public SNS topic up to and including:

2015-07-31 05:00 UTC

and successful chimes resumed again at:

2015-07-31 08:00 UTC

Chimes in between were mostly unpublished, though SNS appears to have delivered a few chimes during that period up to several hours late and out of order.

I had set up Unreliable Town Clock monitoring and alerting through Cronitor.io. This worked perfectly and I was notified within 1 minute of the first missed chime, though it turned out there was nothing I could do but wait for AWS to correct the underlying issue with SNS.

Since we now know SNS has the potential to fail in a region, I have launched an Unreliable Town Clock public SNS Topic in a second region: us-west-2. The infrastructure in each region is entirely independent.

The public SNS topic ARNs for both regions are listed at the top of this page:

"I am writing to you about a very disturbing aspect of Windows 10. Specifically, that the update experience appears to have been designed to throw away the choice your customers have made about the Internet experience they want, and replace it with the Internet experience Microsoft wants them to have."

Right, but what about the experiences that Mozilla chooses to default for users like switching to Yahoo and making that the default upon upgrade and not respecting their previous settings ?What about baking Pocket and Tiles into the experience? Did users want these features? All I have seen is opposition to them.

"When we first saw the Windows 10 upgrade experience that strips users of their choice by effectively overriding existing user preferences for the Web browser and other apps, we reached out to your team to discuss this issue. Unfortunately, it didn't result in any meaningful progress, hence this letter."

Again see above and think about the past year or two where Mozilla has overridden existing user preferences in Firefox. The big difference here is Mozilla calls it acting on behalf of the user as its agent, but when Microsoft does the same it is taking away choice?

30 Jul 2015

In order to have more productivity under my environment, as a command line centric guy, I started three years ago to use zsh as my default shell. And for who never tried it, I would like to share my personal thoughts.

What are the main advantages?

Extended globbing: For example, (.) matches only regular files, not directories, whereas az(/) matches directories whose names start with a and end with z. There are a bunch of other things;

Inline glob expansion: For example, type rm *.pdf and then hit tab. The glob *.pdf will expand inline into the list of .pdf files, which means you can change the result of the expansion, perhaps by removing from the command the name of one particular file you don't want to rm;

Interactive path expansion: Type cd /u/l/b and hit tab. If there is only one existing path each of whose components starts with the specified letters (that is, if only one path matches /u/l/b*), then it expands in place. If there are two, say /usr/local/bin and /usr/libexec/bootlog.d, then it expands to /usr/l/b and places the cursor after the l. Type o, hit tab again, and you get /usr/local/bin;

Nice prompt configuration options: For example, my prompt is currently displayed as tov@zyzzx:/..cts/research/alms/talk. I prefer to see a suffix of my current working directory rather than have a really long prompt, so I have zsh abbreviate that portion of my prompt at a maximum length.

The Z shell is mainly praised for its interactive use, the prompts are more versatility, the completion is more customizable and often faster than bash-completion. And, easy to make plugins. One of my favorite integrations is with git to have better visibility of current repository status.

As it focuses on the interactive use, is a good idea to keep maintaining your shell scripts starting with #!/bin/bash for interoperability reasons. Bash is still most mature and stable for shell scripting in my point of view.

So, how to install and set up?

sudo apt-get install zsh zsh-lovers -y

zsh-lovers will provide to you a bunch of examples to help you understand better ways to use your shell.

To set zsh as the default shell for your user:

chsh -s /bin/zsh

Don't try to set zsh as default shell to your full system or some things should stop to work.

Two friends of mine, Yuri Albuquerque and Demetrius Albuquerque (brothers of a former hacker family =x) also recommended using https://github.com/robbyrussell/oh-my-zsh. Thanks for the tip.

How to install oh-my-zsh as a normal user?

curl -L http://install.ohmyz.sh | sh

My $ZSH_THEME is set to "bureau" under my $HOME/.zshrc. You can try "random" or other themes located inside $HOME/.oh-my-zsh/themes.

And, if you use Ruby under RVM, I also recommend to read this:
http://rvm.io/integration/zsh

Because I've run make deb-pkg so many times, I've started to see exactly where it starts to slow down even with really large machines. Observing cpu usage, I noticed that many parts of the build were serialized on a single core. Upon further investigation I found the following.