Well, it’s been a while! Since we last posted a general update, the Launchpad team has become part of Canonical’s Online Services department, so some of our efforts have gone into other projects. There’s still plenty happening with Launchpad, though, and here’s a changelog-style summary of what we’ve been up to.

Launchpad has had Git-to-Bazaar code imports since 2009, along with imports from a few other systems. These form part of Launchpad’s original mission to keep track of free software, regardless of where it’s hosted. They’re also very useful for automatically building other artifacts, such as source package recipes or snap packages, from code hosted elsewhere. Unfortunately they’re quite complicated: they need to be able to do a full round-trip conversion of every revision from the other version control system, which has made it difficult to add support for Git features such as signed commits or submodules. Once one of these features is present anywhere in the history of a branch, importing it to Bazaar becomes impossible. This has been a headache for many users.

We can do better nowadays. As of last year, we have direct Git hosting support in Launchpad, and we can already build snaps and recipes straight from Git, so we can fulfil our basic goal more robustly now with a lot less code. So, Launchpad now supports Git-to-Git code imports, also known as Git mirroring. You can use this to replace many uses of Git-to-Bazaar imports (note that there’s no translations integration yet, and of course you won’t be able to branch the resulting import using bzr).

We just rolled out a new feature for Launchpad’s Git repository hosting: Git-based merge proposals can now be linked to Launchpad bugs. This can be done manually from the web UI for the merge proposal, but normally you should just mention the Launchpad bug in the commit message of one of the commits you want to merge. The required commit message text to link to bugs #XXX and #YYY looks like this:

LP: #XXX, #YYY

This is the same form used for Launchpad bug references in debian/changelog files in source packages, and the general approach of mentioning bugs in commit messages is similar to that of various other hosting sites.

Bugs are not automatically closed when merge proposals land, because the policy for when that should happen varies from project to project: for example, projects often only close bugs when they make releases, or when their code is deployed to production sites.

Users familiar with Bazaar on Launchpad should note that the model for Git bug linking is slightly different: bugs are linked to merge proposals rather than to individual branches. This difference is mainly because individual branches within a Git repository are often much more ephemeral than Bazaar branches.

Documentation is here, along with other details of Launchpad’s Git hosting.

Snap packages

Allow snap package owners to enable/disable unrestricted processors (matches the change made for PPAs last month)

Soyuz (package building)

Launchpad now accepts uploads of binary packages from builders with file modification times up to 24 hours in the future (rather than 8) or as early as 1975 (rather than 1984), to match Debian

Notifications are no longer generated for package copy jobs that in fact end up being no-ops

Also, as previously announced, we and our sysadmins have been putting a lot of work into extending ScalingStack to other architectures and making it more reliable, and as a result we can now offer general access to PPAs for the ppc64el architecture.

If you are a member of Launchpad’s beta testers team, you can now try out webhooks for Bazaar branches and Git repositories. These can be used to set up integration with external sites for various purposes, such as running CI jobs or publishing documentation. We expect to open this up to all Launchpad users soon, but in the meantime please do file a bug against Launchpad itself if you encounter any problems.

Personal package archives on Launchpad only build for the amd64 and i386 architectures by default, which meets most people’s needs. Anyone with an e-mail address can have a PPA, so they have to be securely virtualised, but that’s been feasible on x86 for a long time. Dealing with the other architectures that Ubuntu supports (currently arm64, armhf, powerpc, and ppc64el) in a robust and scalable way has been harder. Until recently, all of those architectures were handled either by running one builder per machine on bare metal, or in some cases by running builders on a small number of manually-maintained persistent virtual machines per physical machine. Neither of those approaches scales to the level required to support PPAs, and we need to make sure that any malicious code run by a given build is strictly confined to that build. (We support virtualised armhf PPAs, but only by using qemu-user-static in an amd64 virtual machine, which is very fragile and there are many builds that it simply can’t handle at all.)

We’ve been working with our sysadmins for several months to extend ScalingStack to non-x86 architectures, and at the start of Ubuntu’s 16.04 development cycle we were finally able to switch all ppc64el builds over to this system. Rather than four builders, we now have 30, each of which is reset to a clean virtual machine instance between each build. Since that’s more than enough to support Ubuntu’s needs, we’ve now “unrestricted” the architecture so that it can be used for PPAs as well, and PPA owners can enable it at will. To do this, visit the main web page for your PPA (which will look something like “https://launchpad.net/~<person-name>/+archive/ubuntu/<ppa-name>”) and follow the “Change details” link; you’ll see a list of checkboxes under “Processors”, and you can enable or disable any that aren’t greyed out. This also means that you can disable amd64 or i386 builds for your PPA if you want to.

We’re working to extend this to all the existing Ubuntu architectures at the moment. arm64 is up and running but we’re still making sure it’s sufficiently robust; armhf will run on arm64 guests, and just needs a kernel patch to set its uname correctly; and powerpc builds will run in different guests on the same POWER8 compute nodes as ppc64el once we have suitable cloud images available. We’ll post further announcements when further architectures are unrestricted.

Soyuz (package building)

Allow overriding the build version for live filesystem builds (#1496074)

Archive owners can change the “build debug symbols” and “publish debug symbols” settings on their own archives, rather than needing to ask an admin

The publisher no longer sometimes tries to update Contents files for immutable suites such as the release pocket of stable series, which could cause checksums in Release to get out of sync (#1448270)

Publishing PPAs now creates clearsigned InRelease files to improve reliability of updates (#804252); we intend to add this to the primary Ubuntu archive as well once one last piece of mirroring infrastructure is made ready for it

Archive owners can enable or disable unrestricted architectures on their own archives, rather than needing to ask an admin; we will unrestrict further architectures once they have adequate virtualised build infrastructure available

Mail notifications

Our internal stakeholders in Canonical recently asked us to work on improving the ability to filter Launchpad mail using Gmail. The core of this was the “Include filtering information in email footers” setting that we added recently, but we knew there was some more to do. Launchpad’s mail notification code includes some of the oldest and least consistent code in our tree, and so improving this has entailed paying off quite a bit of technical debt along the way.

Bug notifications now log an OOPS rather than crashing if the SMTP server rejects an individual message (#314420, #916939)

Recipe build notifications now include an X-Launchpad-Archive header (#776160)

Question notification rationales are now more consistent, including team annotations for subscribers (#968578)

Package upload notifications now include X-Launchpad-Message-Rationale and X-Launchpad-Notification-Type headers, and have more specific footers (#117155, #127917)

Package build infrastructure

Launchpad now supports building source packages that use Debian’s new build profiles syntax, currently only with no profiles activated

Launchpad can now build snap packages (#1476405), with some limitations; this is currently only available to a group of alpha testers, so let us know if you’re interested

Builders can now access Launchpad’s Git hosting (HTTPS only) in the same way that they can access its Bazaar hosting

All amd64/i386 builds now take place in ScalingStack, and the corresponding bare-metal builders have been detached pending decommissioning; some of the newer of those machines will be used to further expand ScalingStack capacity

We have a new ScalingStack region including POWER8-based ppc64el builders, which is currently undergoing production testing; this will replace the existing POWER7-based builders in a few weeks, and also provide virtualised build capacity for ppc64el PPAs

We’ve fixed a race condition that sometimes caused a user’s first PPA to be published unsigned for a while (#374395)

Miscellaneous

The project release file upload limit is now 1 GiB rather than 200 MiB (#1479441)

We spent some more time supporting translations for the overlay PPA used for current Ubuntu phone images, copying a number of existing translations into place from before the point when they were redirected automatically

Code

We fixed a regression in the wrapping layout of side-by-side diffs on bazaar.launchpad.net (#1436483)

Various code pages now have meta tags to redirect “go get" to the appropriate Bazaar or Git URL, allowing the removal of special-casing from the “go" tool (#1465467)

Merge proposal diffs including mention of binary patches no longer crash the new-and-improved code review comment mail logic (#1471426), and we fixed some line-counting bugs in that logic as well (#1472045)

Links to the Git code browsing interface now use shorter URL forms

We’ve also made a fair amount of progress on adding support for triggering webhooks from Launchpad (#342729), which will initially be hooked up for pushes to Git repositories. The basic code model, webservice API, and job retry logic are all in place now, but we need to sort out a few more things including web UI and locking down the proxy configuration before we make it available for general use. We’ll post a dedicated article about this once the feature becomes available.

Notifications about duplicate bugs now include an X-Launchpad-Bug-Duplicate header (#363995)

Package build failure notifications now include a “You are receiving this email because …” rationale (#410893)

Package build infrastructure

The sbuild upgrade last month introduced some regressions in our handling of package builds that need to wait for dependencies (e.g. #1468755), and it’s taken a few goes to get this right; this is somewhat improved now, and the next builder deployment will fix all the currently-known bugs in this area

In the same area, we’ve made some progress on adding minimal support for Debian’s new build profiles syntax, applying fixes to upload processing and dependency-wait analysis, although this should still be considered bleeding-edge and unlikely to work from end to end

We’ve been working on adding support for building snap packages (#1476405), but there’s still more to do here; we should be able to make this available to some alpha testers around mid-August

Miscellaneous

We’ve arranged to redirect translations for the overlay PPA used for current Ubuntu phone images to the ubuntu-rtm/15.04 series so that they can be translated effectively (#1463723); we’re still working on copying translations into place from before this fix

Projects and project groups no longer have separately-editable “display name” and “title” fields, which were very similar in purpose; they now just have display names (#1853, #4449)

Cancelled live file system builds are sorted to the end of the build history, rather than the start (#1424672)

Users of some email clients, particularly Gmail, have long had a problem filtering mail from Launchpad effectively. We put lots of useful information into our message headers so that heavy users of Launchpad can automatically filter email into different folders. Unfortunately, Gmail and some other clients do not support filtering mail on arbitrary headers, only on message bodies and on certain pre-defined headers such as Subject. Figuring out what to do about this has been tricky. Space in the Subject line is at a premium – many clients will only show a certain number of characters at the start, and so inserting filtering tags at the start would crowd out other useful information, so we don’t want to do that; and in general we want to avoid burdening one group of users with workarounds for the benefit of another group because that doesn’t scale very well, so we had to approach this with some care.

If you check “Include filtering information in email footers”, Launchpad will duplicate some information from message headers into the signature part (below the dash-dash-space line) of message bodies: any “X-Launchpad-Something: value” header will turn into a “Launchpad-Something: value” line in the footer. Since it’s below the signature marker, it should be relatively unobtrusive, but is still searchable. You can search or filter for these in Gmail by putting the key/value pair in double quotes, like this:

At the moment this only works for emails related to Bazaar branches, Git repositories, merge proposals, and build failures. We intend to extend this to a few other categories soon, particularly bug mail and package upload notifications. If you particularly need this feature to work for some other category of email sent by Launchpad, please file a bug to let us know.

Today we’re announcing early support for hosting Git repositories directly on Launchpad, as well as or instead of Bazaar branches. This has been by far the single most commonly requested feature from Launchpad code hosting for a long time; we’ve been working hard on it for several months now, and we’re very happy to be able to release it for general use.

This is distinct from the facility to import code from Git (and some other systems) into Bazaar that Launchpad has included for many years. Code imports are useful to aggregate information from all over the free software ecosystem in a unified way, which has always been one of the primary goals of Launchpad, and in the future we may add the facility to import code into Git as well. However, what we’re releasing today is native support: you can use git push to upload code to Launchpad, and your users and collaborators can use git clone to download it, in the same kind of way that you can with any Git server.

Our support is still in its early stages, and we still have several features to add to bring it up to parity with Bazaar hosting in Launchpad, as well as generally making it easier and more pleasant to use. We’ve released it before it’s completely polished because many people are clamouring to be able to use it and we’re ready to let you all do so. From here on in, we’ll be adding features, applying polish, and fixing bugs using Launchpad’s normal iterative deployment process: changes will be rolled out to production once they’re ready, so you’ll see the UI gradually improving over time.

What’s supported?

You can:

push Git repositories to Launchpad over SSH

clone repositories over git://, SSH, or HTTPS

see summary information on repositories and the branches they contain in the Launchpad web UI

follow links from the Launchpad web UI to a full-featured code browser (cgit)

push and clone private repositories, if you have a commercial subscription to Launchpad

propose merges from one branch to another, including in a different repository, provided that they are against the same project or package

What will be supported later?

Launchpad’s Bazaar support has grown many features over the years, and it will take some time to bring our Git support up to full parity with it and beyond. Git repositories use a somewhat different model from Bazaar branches, which we’ve had to account for in many places, and some facilities will require complete reimplementation before we can support them with Git.

Here’s an incomplete list of some of the features we hope to add:

useful subscriptions (currently only attribute change notifications work, which are not usually very interesting in themselves)

RSS feeds

mirroring

webhooks

an integrated code browser

See our help page for more known issues and instructions on using Launchpad with Git.

Launchpad is free software, licensed under the GNU AGPLv3. We’d be very happy to mentor people who want to help out with parts of this service, or to build things on top of it using our published API. Some preliminary documentation on this is on our developer wiki, and you can always contact us for help.

Launchpad’s bug and branch privacy features were replaced by information sharing that permits project maintainers to share kinds of confidential information with people at the project level. No one needs to manage bug and branch subscriptions to ensure trusted users have access to confidential information.

The Disclosure features

Disclosure is a super feature composed on many features that will allow commercial projects to work in private. Untrusted users cannot see the project’s data. Project maintainers can share their project with trusted users to reveal all or just some of the project’s data. The ultimate goal is to create private project in Launchpad, but that feature required several other features to be completed first. The Purple squad worked on Trusted Pickers, Privacy Transitions, Hardened Projects, Social Private Teams, and Sharing.

There was a lot of overlap between each feature the Purple squad worked on. Though we could start each feature independent of one another, we could only complete about 90% of each. When the Sharing UI changes entered beta, we were unblocked and fixes about most of the remaining issues, but fixing all the issues required all projects to switch to Sharing. We did not consider Sharing, or any of the required features complete until we fixed all the bugs.

Disclosure facts

Planning started in June 2010 to replace the existing privacy mechanisms with something that would scale.

Early testing revealed that users did not trust Launchpad because the UI could not explain what was confidential, or what the consequences of a change would be — this needed to be fixed too.

149 related bugs were identified in Launchpad.

Work started in June 2011 by the Purple squad.

Replacing the old privacy mechanisms and addressing the trust and information issues took 16 months.

About 45,000 lines code were added to support the features.

About 15% of the lines were for missing JavaScript test coverage.

More that 700 bugs were fixed in total.

About 5% of the fixed bugs were caused by the old non-scaling privacy mechanisms.

About 4% of the fixed bugs were caused by old JavaScript enhancements that broke features for non-JavaScript users.

Lessons learned

Misrepresentation of what is confidential, or what will be confidential or public is very important to users — more important than supporting private data.

Privacy/Sharing must be a first-class mechanism beneath all the mechanisms that work with confidential data.

Privacy was added on top of bugs, and it failed to scale to 100’s of bugs.

Privacy was added on top of branches, and it failed to scale to 1000’s of branches.

Filtering private items in code, or in database joins is not fast enough to work with 100,000’s of items.

Launchpad’s ReSTful object API is not suitable for working with large collections of objects like bugs or branches; a lighter, service-based approach was used to quickly work with large amounts of data.

Users need to work with confidential data via the API, using a text web browser from servers, using a browser with accessibility tools, as well as the common case of using a JavaScript enabled browser.

Lots of mock-ups and interactive tests will not predict all the interactions a user will have with real data; test with real code and data early to developer the final design.

Launchpad’s bug and branch privacy features are being replaced by information sharing that permits project maintainers to share kinds of information with people at the project level. No one needs to manage bug and branch subscriptions to ensure trusted users have access to confidential information.

Maintainers can share and unshare their project with people

Project maintainers and drivers can see the “Sharing” link on their project’s front page. The page lists every user and team that the project shares with. During the transition period of the beta, you might see many users with “Some” access to “Private Security” or “Private” user information. They have this access because they are subscribed to bugs and branches. Maintainers can unshare with users who do not need access to any confidential information, or just unshare a bug or branch with a user. Maintainers can share share with a team to give them full access to one or more kinds of confidential information.

I have prepared a video that demonstrates the features (my apologies for the flickering)

Commercial projects can set bug and branch policies

Projects with commercial subscriptions can also change bug and branch sharing policies to set the default information type of a bug or branch, and control what types they may be changed to. Maintainers can set policies that ensure that bugs and branches are proprietary, and only proprietary, to ensure confidential information is never disclosed.

Sharing can be managed using API scripts

I maintain many project which have a lot of private bugs and branches. The sharing page lists a lot of people, too many to read quickly. I know most work for my organisation, but I don’t even know everyone in my organisation. So I wrote a Launchpad API script that can be run by any project maintainer to share the project with a team, then unshare with the team members. The members still have access to the bugs and branches and their subscriptions still work, but they will lose access to my project when they leave the team. This arrangement makes it very easy to manage who has access to my projects. share-projects-with-team.py is run with the name of the team and a list of projects to share with it.

Project maintainers can now see all the private bugs in their project. While Launchpad tried to ensure the proper people could see private bugs in the past, the old subscription mechanism was brittle. Users could unsubscribe themselves and lose access, or retarget a bug to another projects which does not update bug subscriptions. The Purple squad migrated project configurations to project sharing so that all private information was shared with project maintainers. Project sharing ensures that confidential information is disclosed to the proper people.

If you are a project maintainer, you might be surprised to find old private bugs that you have never seen before. This happened to me. Some ancient private bugs were in the “New” listing of bugs, other were buried in search results. You can search for just private bugs to review all private bugs.

Privacy terminology is restored

We reverted the information type terminology changes introduced a few months ago.

User data ? Private

Embargoed Security ? Private Security

Unembargoed Security ? Public Security

While the jargon-laden terms helped the small number of people who work with confidential information, the people who report bugs were confused. The most common reason for unwanted disclosure is that people enter confidential information, and cannot see how to make it private. Sometimes a user may not notice the mistake until a few minutes later. We also revised the descriptions of the information types to help new users quickly select the correct information type.

You can now hide your own bug and question comments. If you want to hide a comment made in error, you can use the “Hide comment” action.

You can see it, and even unhide it if you choose. The project’s maintainer or the trusted people delegated to work with private information can still see your comment.

This allows you, or the people the project shares private information with, to hide just the comments that contain personal information. The bug does not need to be made private if the comment can be hidden. Project maintainers can also hide comments because they contain spam or abuse.

The Purple squad recently updated bug reporting and searching to understand the new privacy rules. Some of the changes were requirements to support sharing, others were opportunities we took advantage of.

Improvements to bug reporting and forms

The Purple squad updated the bug reporting UI to make it consistent with the bug pages. We choose to develop one consistent and tested UI rather than update the many kinds of widgets used in bug forms.

Undecided is the first importance because it is the default importance.

Improvements to bug searching

Advanced bug search was updated after we discovered that recent changes made it possible fix some long standing issues with a few additional lines of code.

Anyone can search for Private or Embargoed Security bugs that are shared with them.

Autocomplete works with bug tags.

Usability and Accessibility fixes

We discovered that the popups that show bug status, importance and information type did not work with keyboards. It was possible to tab out of every other kind of popup by accident. We made deep fixes to the code so that all launchpad popups work with keyboard.

All users can now see the information type section that replaces the privacy and security section shown on bug and branch pages. This change allows users to clearly state the kind of information a bug or branch contains. Launchpad will soon permit project maintainers to share information types instead of managing individual bug and branch subscriptions. Project maintainers can see a link on their project’s front page to the ”Sharing” page. Sharing lists all the users and teams their project shares some private bugs and branches with. This list might be surprising. Launchpad Beta testers will soon be able share and unshare kinds of information to simplify management of whom the project discloses private information to.

Launchpad beta testers are seeing information types on bug reports. Launchpad replaced the private and security checkboxes with an information type chooser. The information types determine who may know about the bug.

When you report a bug, you can choose the information type that describes the bug’s content. The person who triages the bug may change the information type. Information types may also change as a part of a workflow, for example, a bug may start as Embargoed Security while the bug is being fixed, then the release manager can change the information type to Unembargoed Security after the release.

Testing in phases

In the first phase of this beta, Launchpad continues to share private information with bug supervisors and security contacts using bug subscriptions. The project maintainer may be managing hundreds of bug subscriptions to private bugs, and people are getting unwanted bug mail.

In the second phase of the beta, the project maintainer can share information types with people…the maintainer is only managing shares with a few teams and users and people are not getting unwanted bug mail.

Watch the video of information types and sharing to see the feature in use and hints of the future.

Launchpad beta testers will now see the descriptions of bug status and importance when making updates to the bug page. Launchpad pickers can now show the descriptions of the options you can choose.

Launchpad’s rules for defining a list of options you can choose have always required descriptions, but the only places you could see them were in some forms where they were listed as radio buttons. Bug status and importance where never shown as radio buttons, so their description were only know to people who read Launchpad’s source code. Users need to see the descriptions so that there is a common understanding of terms that allows us to collaborate. The original bug importance descriptions were written in 2006 and only made sense for Ubuntu bugs. We revised the descriptions for the improved picker.

There has been a lot of confusion and disagreement about the meaning of bug statuses. Since users could not see the descriptions, we posted the definition on help.launchpad.net. Separating the status description from the status title did not end the confusion. We revised the descriptions for the improved picker, but I think we need to make more changes before showing this to everyone. The picker appears to rely on colour to separate the choice title from description. Not all choices will have a special colour, and in the case of bug status there are two choices that appear to be the same grey as the description text:

The picker enhancements were made for the disclosure feature. We are changing the presentation of bug and branch privacy to work with the forthcoming project sharing enhancements. Early testing revealed that users need to know who will be permitted to see the private information when the bug is changed. This issue was similar to the long standing problem with bug status and importance. We decided to create a new picker that solved the old problem, that we could then reuse to solve the new problem.