Blog Post

Apt URL Part Two

OK, so after going through comments on my previous post about Apt URL it has become obvious to me. Apt URL is a band aid more than it is a way for people to easily distribute software. Is this a bad thing? No, not in my personal opinion. It seems the main arguments I received in the post, and especially on IRC (thanks to all of you who messaged me bitching me out, that rocks!) as as follows:

Once Ubuntu is released, we aren’t getting new updates.

Package so-and-so hasn’t been updated in 2 years

This will allow software developers to get their software out to more people.

OK, so it is obvious why I call Apt URL a band aid, and points #1 and #2 show this. For point #1 it is obvious that Backports aren’t getting utilized as they probably should. Point #2 shows us that there are more merges on MoM than there are developers to handle that, and that there is a ton of software we aren’t paying attention to. This is something that has to be fixed, but has proven difficult for the past few years.

That brings me to point #3. In a comment in my last post, Skype was brought up, and how it isn’t in the Ubuntu repositories. Is there a reason that Skype can’t go into Multiverse or the Canonical repositories? Is there something I am missing when it comes to the non-free repositories? I will admit I do not follow them since I attempt to keep my system RMS happy :p

OK, so here is my other question, slash, problem. Security! I keep hearing about this “whitelist.” Am I to expect that people are going to go through the Core Developer process in order to get on this so-called whitelist? If you don’t do a process like this, well you just flat out disrespected every MOTU and Core Developer in our community. If you make them go through a process like this, then why can’t they be a MOTU or Core Developer in Ubuntu? This is my big issue really. If you don’t make them go through the process that every MOTU and Core Developer has done then you might as well spit in those people’s faces who have put their blood, sweat, and tears into gaining a certain level of trust. And if you do make them go through the same process, then what the heck, it makes no sense.

I am still looking for solid information on why this is good, and how it can be utilized for something other than a band aid. Martin Owens had my favorite comment on the previous post, about what kind of society do I think we live in and what not. Martin, we live in a society right now where people need protection more than anything. I am not talking about the old G-Dub terrorist protection plan, I am talking about those evil little kids in mommy and daddy’s basement using other people’s scripts to do damage. Linux, just like Windows and Mac, is as secure as its user. I think it is in Ubuntu’s best interest to protect the users as much as possible, but not to the point where we cut off their freedoms. If people want Apt URL, give it to them, but I think Ubuntu should make the same statement it did about Automatix years back.

If we want to make it easy for people to get the latest and greatest software, then we need to start working on fixing our infrastructure so we can do it correctly and safely. Since there is no single package manager to rule them all, Linux software distribution will continue to be a pain in the ass. Here is an idea. How about a mailing list or such, where upstream developers can announce new software, updated software, and what not? Everyone who wants to be a packager, look there and get to work? There has to be a way to have solid upstream <—> downstream communications, it is sounding like it isn’t happening to me.

Merges….can you pin down what the merge workload looks like? Can you historically trend it as a function of time say weekly merge workloads over the past 2 years? If the workload per human being with the ability to deal with merges is trending upward over time, you need to take a real hard look at why merges are a growing backlog in the process. And if the workload isn’t trending upwards and has been pretty stable or oscillating over the last two years, you’ll need to look elsewhere for the underlying problem.

The other hard question that you have to take a look at is why ppas are being heavily utilized instead of backports. Is the qualification process to be a package maintainer in Ubuntu too high a bar to meet initially? Can that be restructured? What you might want to do is do a statistical survey of very active ppa users in launchpad, people doing lots of ppa package work relative to the average but who have not made an MOTU level commitment to Ubuntu. Identify that group of motivated packagers and ask them as a group why they aren’t going through the process of being a MOTU member. Those people are the motivated workers, they are your best recruitment target for MOTU..figure out why they aren’t participating and adjust the process accordingly.

And beyond that, is there a need for a new workflow that can take the work being done to build ppas to funnel packages into backports when appropriate?
Can launchpad flag PPAs based on a set of popular active PPAs for MOTU members to pull from to popular backports?

There is already a way for packagers to be automatically notified by new upstream releases – in Debian packaging, it’s the debian/watch file. In fact, there’s a list of packages out of sync with upstream here: http://dehs.alioth.debian.org/no_updated.html

It seems that packagers just needs to be made aware of these tools, and start using them. New packages that aren’t yet in the tree will still have this problem, but with more MOTUs and more cooperative software providers, that’s not a problem at all – once a package is in tree, it just needs to be bumped occasionally.

For point #1 it is obvious that Backports aren’t getting utilized as they probably should.

Sometimes there are just too many complexities for a backport, but often times it’s because a developer or packager is using minimum requirements that are unnecessarily high (like defaulting to the version in Jaunty or Ibex) instead of true minimum requirements. I’ve taken packages that said they require version Y, edited the control file to require version X, compiled, installed, and never had problems. I discuss this more in this blog post which was inspired by this post.

As a side note, I’m wondering what’s the recommended way for third-party developers to distribute their software to Ubuntu users.

As far as I can see, the only “supported” way is to get the software into universe or multiverse archives, which is far too much work for spare-time developers. Also, it seems this doesn’t allow to publish software for older Ubuntu releases like Hardy.

Putting the software into a PPA is a better way at first glance, but then again from-scratch packaging and targeting different Ubuntu releases doesn’t seem to be documented well.

Which leaves the bad old way of distributing links to tgz files or shells scripts or .run files, which install and overwrite files all over the disk, but at least it’s simple to do for the developer and doesn’t show obvious problems for the user.

I’m asking because I still have about three little tools around which I currently keep as tgz + “installation instructions” but would like to distribute as debs. What am I to do with these?

@Jef – awesome response! As for getting that information, I would have to speak with Launchpad devs, but this is a very good point that needs to be researched, if it hasn’t been already. As for the process of being a MOTU, it is really easy actually. As long as you can package, have a sustained contribution to packaging and a good community presence, then MOTU is pretty much a given. Core developer is of course a bit more difficult due to higher standards. What goes in Main can potentially be destructive for our users. There has always been a large amount of merges, and I remember back 3 to 4 years ago, a lot of developers concentrated on merges at the beginning of each cycle and then towards the freeze where merges and syncs were to be stopped. Since that time, I think the amount of community developers seems to have shrunk due to people having conflicts with their personal lives.

@Joshua – One thing I have noticed, is if you go back over a year ago, almost 2, we kept debhelper at the same version just to make backports easier. Since then I am seeing people bump their debhelper versions in the latest development cycle, and this can of course hurt backporting to a certain extent. If we weren’t to concentrate on backporting a latest release to LTS unless necessary, then there shouldn’t be that much of a complexity.I think right now, debhelper version 7 goes back to Hardy, but I could be mistaken. As for the preview button, I swore I had one just a month or so ago, but that could have been on a beta blog layout I was working on. You are right and it is something I am working on. Hopefully I will have the same layout as my main website so everything is uniform. I apologize for not having the preview button at this time though.

@yman – Hopefully you weren’t utilizing those 2 links in support of Apt URL because Apt URL wouldn’t do anything for either of those 2 topics. As for #20129 CentOS does the same. I don’t see why we couldn’t do this for Debian/Ubuntu utilizing /etc/lsb_release, as that is almost exactly how CentOS does it as well. As for #13004 I read your one comment about that situation with pinning being a huge hassel, and you are right, but I think it is a huge hassle for a reason. People who would be interested in working a system similar to what was described in that topic would 99.9% of the time not be a new user. I think making it difficult makes sure that a new to Linux user doesn’t hose a system and then freaks out on IRC about it, but I could be wrong.

I also noticed in #13004 the site getDeb being used as an example. I know at one time getDeb packages were created with checkinstall, the same way some packages in Automatix were created. People would go to install this and a lot of the time it would hose up just the application if they were lucky. I think getDeb has become a bit more real in a packaging sense since then, but I am not sure about that either. This whole situation concerning packaging is a real pita if you ask me.

I also said I am tired of wasting time on Apt URL, and here I am with 2 blog posts about it. How come nobody has called me out yet? 🙂 I am such a hypocrite :p

Thanks guys for the comments, they have made me want to look in to a few things now.

Thomas

There are two good reasons why Skype is not in the repositories: it is closed source, and it may be evil. I am not saying that it is – although without the source there is no way of knowing. It certainly is suspicious.

Now I guess it could go into the Canonical repository, if Canonical gets some (secret?) reassurance, but it seems that there is a potential conflict of interest even there.

The conclusion is that closed source commercial software is probably best out of the official repositories. You can always add another repository. Yes, this should be made easier, but I am not sure what the best way would be. A GUI for repository management? Should that be integrated with package management? Some usability expert please elaborate 🙂

“I also noticed in #13004 the site getDeb being used as an example. I know at one time getDeb packages were created with checkinstall, the same way some packages in Automatix were created.”

Here we are again with the FUD based comparison to Automatix for everything that is not on universe. GetDeb started as a personal googlepages site providing checkinstall based packages, yes. This was 3 years ago, such approach was used for a few weeks due to my lack of know-how at that time, it then moved to a team project, with a self-developed CMS for the site, and it’s own automated build system.

Your concerns about the PPA’s and 3rd party providers is clearly based on lack of information. You really need to get out of the “universe” hole to view the entire world before commenting on other’s people work and about software distribution in general.

@Thomas: The purpose of the multiverse repository is for packages that are closed source and/or have other restrictions that limit modification and distribution. I believe submitting to multiverse is slightly more difficult than submitting to universe because of the complications that surround this.

Skype is a perfect example of this. The reason why Skype cannot be in multiverse is because the Skype licence explicitly forbids it. For example:

“1.3 You will not distribute other products or services together with Skype Software, unless You are a publisher of computer magazines for end users and distribute the Skype Software with Your magazine(s) for free.”

@João Pinto – FUD? You are crazy dude. I know getDeb has gotten better, but in the past it was hosing machines just like Automatix was. I can’t think of anything else to compare this entire situation to. Yes, I really do not trust 3rd parties either. By the way, I am not stuck in the universe, as a lot of my work goes on in main anyways :p My concerns are not based on lack of information, and I think that was a stupid remark based on a lack of information. If you read at all my previous post, I clearly stated that offering Apt URL support on a team’s PPA (ie. Desktop Team, Kubuntu Team, Server Team, etc) would be the safest way to initially offer Apt URL. As for personal PPAs that would really depend on the person. I feel that person should be a MOTU or a Core Developer before they are “whitelisted.” Here is a question for you, since I don’t use getDeb. How come you do not package for Ubuntu? Your experience is obviously great and would benefit Ubuntu by contributing directly.

@SeanHodges – ah ha! That’s why it is not in the repos. Thanks for that!

Jef Spaleta

@nixternal:

Try not to rely on personal mermory, or collective institutional memory about trending, actually having a graph that trends a workload metric in front of you for is far more reliable..especially if its highly fluctuating with a long term upward trend across releases. The fluctuations are going to warp perception. Fight the tendency to ballpark it, and find a way to trend workload.

“Easy” is such a fluid definition. You may think the MOTU process is easy enough, but do the potential candidates who are active as PPA packagers think it is? The definition of easy could have slipped a little bit as the demographics of your contributor community changed over time. Seeing if the active PPA packagers think its easy is the only way forward here. It could just be a lack of communication about the role of MOTU. You might find out that its easier to know about PPAs, then it is about knowing about MOTU. That makes some sense, as technically launchpad is a general purpose hosting service not tied to the Ubuntu distribution release policies..and PPAs are part of that service. You don’t see MOTU membership communicated through general launchpad interaction like you do PPAs. In a sense Launchpad nature has made PPAs easyer to discover and thus easier to use.

Are you able to track volunteer MOTU activity levels? Do you have a sense as to how many active individuals are doing the day-to-day work or release-to-release work? And you have to be careful about setting workload expectations for volunteers and keeping a track of volunteer activity without being judgemental. You have to build in an expected turnover rate for volunteers into any process which is heavily dependent on them. You should absolutely expect volunteer MOTUs to step back after sometime, but have you generated an expectation on that timeframe (in an average sense) and built a process which anticipates losing volunteers at an average rate? How has the MOTU group grown over time? Was it impulsive growth with a large chunk of the team built in a burst? Or has it been slow aggregation? Impulsive growth can have draw backs on the back end because you should then expect that group of people to stepback around the same time (on average) so you lose people impulsively too…and institutional process memory with them.

@Jef – you are right about easy, but if someone can spend time packaging it up and putting it into a PPA or other repo, it should be easy for them to make MOTU. If they are having problems making MOTU, then I don’t want to use their repos or packages. In a way MOTU would be easier, as you don’t have to mess with your PPA, nor do you have to figure out how dput works for the PPA. The fact that you have to search or discover a PPA makes it more difficult than the standard set of repos you are provided after installing Ubuntu. The PPAs are typically utilized for testing prior to sending stuff to backports. I know the Kubuntu team is relying more on backports now than they are the PPA. This time around with KDE 4.3 beta release, we will use the PPA as it is a beta release, but when 4.3 is stable, then we will get it backported.

Is that a statistically valid statement? That might be the intent of PPAs when they were first created..but is that the typical usage case now? I don’t think the Apt URL discussion would be so…controversial….if the typical PPA workflow really was as a feeder to Backports. I think the actual PPA usage has grown into a new role while perception of intended usage hasn’t changed. I think its the dissonance that is causing people to look for Apt URL whitelisting as a technical solution..to what is fundamentally a breakdown on package policy/workflow. I think you have to throw away all the existing assumptions about how PPAs are meant to be used, and look at how they are actually being used..and then work forward from there.

There are a crapload of individual ppas. Is the pre-backports workflow “typical” at this point? 9489 registered Ubuntu PPAs at this point over a 1000 “active” whatever that means. Is pre-backport testing the dominate usage scenario? Can you actually trend the flow of packages from PPA to backports? Maybe there is a difference between the flow rate for main as compared to universe for example. This might be hard to discover if there is no direct linkage in launchpad between PPA and backports that can be logged. If PPAs really are meant to feed into Backports, finding a way to instrument that feeder process as part of launchpad workflow would make sense and give you something to trend accurately.

Another question, can you get hard number of which PPAs and which binaries inside of PPAs are the most popular. Are the most popular binaries going into backports in a timely manner? PPAs aren’t mirrored are they? If they aren’t then someone should have access to logs showing PPA binary download activity. which could be mined. Are the most popular PPAs pre-backport testing or have they effectively a parallel repository structure?

You should probably stop talking to me now. You’ve given me about 5 specific new talking points about the benefits of streamlining PPA workflow by opening Soyuz. I will end up quoting you next time I have the opportunity to debate a Canonical rep about the closed status of Soyuz.

Subscribe to nixternal.com

Archives

Who is Richard A. Johnson?

My name is Richard A. Johnson. I am a professional developer, as
well as an open source enthusiast, advocate, and author. I spend
any free time I can get cycling.

Back in 1994 I was introduced to the world of open source
software by installing Linux on a PC I had purchased from the
Navy Exchange. At that time I had no idea what I was going up
against, but never once did I let myself get frustrated, as
Windows had already done that to me. Since then, I have
completely immersed myself into the open source world and have
been an active contributor.