Listaller Project – To infinity and beyond!

Nearly everyone who’s reading this blog will know about Listaller, a project started by me to make cross-distro 3rd-party software installations possible & secure.

This project has been started years ago, and lots of things have changed in the Linux desktop world and down the stack, so I think it is time to look back now and summarize what has happened – and also to finally write down my evil plan about what Listaller should achieve. People who know me and who I’m working with already know what I would like to achieve and pushed me a little to write this “vision” down.

I’m not a big fan of project “visions”, but in this case to define a final goal visible to everyone (and not just me) will help a lot.

So, if you want to read something about the state of the Linux desktop and my view on the recent “Linux desktop has failed” discussion as well as lots of history and future visions of the Listaller Project, keep on reading! (If not, go here and enjoy the day ;-))

The Past

When Listaller was started in December 2007 (I guess, I don’t have an exact date), software management on Linux distributions was in a very sad state. It was usual to manage software in distributions using a native package manager, like Synaptic or YaST. Also, extending the software repository was not really easy. Ubuntu did it’s first experiments with an application-centric software manager that time and PPAs weren’t that common. (I don’t know if Launchpad already offered this service back then – I think it was implemented later)

Regarding cross-distro 3rd-party software installing, there were a few competing projects like LOKI/MoJo, Autopackage, ZeroInstall, Klik, etc. which all weren’t that common and only a few people used that software. Also, they didn’t solve the main issue: Packages aren’t user-friendly! A “package” is a somewhat abstract concept which is hard to understand for non-technical users. These projects only focused on installing cross-distro packages, but I wanted more.

So I started the Listaller project. Listaller was designed to build a cache of applications installed on a distribution and offer uninstalling these apps, no matter how they were installed. Listaller implemented it’s own package-manager abstraction layer, which basically worked with plugins which then called native distribution tools like “dpkg”, “rpm”, “apt-get” or “yum”. The only distributions supported were Debian, Ubuntu, SUSE and we had very poor support for Fedora.

Listaller also contained a new format to install applications, which was designed to abstract package management complexity. The Listaller package format was able to carry links to existing native packages and download & install them in order. It also was able to contain any binary installer, so it could also carry Autopackage or LOKI installers and perform necessary steps to execute these binaries and make sure everything is set up correctly. Finally, it was also possible to include the application install files directly into a Listaller package (file extension *.lipkg at the beginning, later changed to *.ipk) – It was planned that this solution should later be the only possible way to install software, the other IPK-package-modes were made to ease transition. The Listaller package generator also produced a logobutton, showing all distributions this package supported, which developers were able to put on their websites, because the generated Listaller packages were sometimes still limited to some distributions. (Listaller was able to detect that to a certain degree) I found only one of these buttons on my disk. It looked like the thing on the right side.

The software manager was very similar to later tools managing applications instead of packages, which was the goal of this effort.

Here’s a screenshot of the very early Listaller application manager (and it’s crappy UI):

With the rise of PackageKit, the whole situation shifted a little: Now we finally had one abstraction layer for all package managers distributions offered, which was much more powerful than the poor thing I created. I followed the PackageKit project since version 0.1.18 (I think) and later joined the project as developer and implemented support for it in Listaller.

During that time, the Klik installer project died and Autopackage merged into Listaller (after both projects first only did collaboration), leaving only very few projects with cross-distro software management in scope.

The the AppStream project was created (unfortunately I wasn’t able to attend the meeting due to exams) and suddenly all points the Listaller Project wanted to achieve back then had been solved – except for one: Cross-distro software installations. (AppStream implements the app-centric software management in a cross-distro way, so this Listaller goal is now achieved)

So I shifted the scope of Listaller to 3rd-party software-installations only and started a new implementation of it from scratch using the new Vala language. I also extended PackageKit with support for Listaller, because having multiple UIs for package management is highly confusing for users and the new design of Listaller made it possible to extend any PackageKit GUI with the ability to manage Listaller packages.

Listaller was extended to use ZeroInstall feeds to satisfy dependencies too, so resolving dependencies would work better, and both ZeroInstall and Listaller could benefit from the availability of more feeds. The Listaller support in PackageKit support was then split out into a plugin, so distributions are able to enable/disable Listaller as they like. (and also the plugin code made PackageKit’s code much better and easier to handle!)

Many other changes happened too, which I won’t summarize, this blogpost is already too long 😉 – Unfortunately the rewrite made Listaller unusable for a very long time and some people already considered the project dead.

The Present

So, many people might now ask the following:

Why do we need cross-distro packages?

I consider cross-distro software management a key element for the success of Linux desktops.

At time we have a very old protocol how software reaches end-users: Upstream creates it, downstream the distributions package it and deliver it to their users. This unfortunately does not leave any choice for people to selectively install or update the software they want. It is also not easily possible to install software which is not in the repositories. Of course, someone could create packages – but then he has to build his software for a multitude of distributions, which is lots of work (you can’t even target all RPM-based distributions using just one spec-file without doing many distro-specific checks)

It is important that developers can get quick feedback from their users on new software, without needing to go into the details of package management and having to wait for distributors to ship new software. (Take Debian stable as an example) Also, users should be able to install software without having to worry about compromising their system. (By installing a foreign package, you’re basically giving someone else root access to your computer)

The OpenBuildService, although being a great tool, is not a solution for this – also PPAs aren’t, because any change on the native package configuration can make the system unusable, install malicious software or break upgrade paths, and they’re hard to handle for non-technical users.

Where are the app developers?

There is a big lack of commercial software for Linux. Why? Usually we hear the argument that “Linux has not a big enough market share”. I don’t think this argument counts that much, as Apple also had a very small market share and they had commercial apps too back then. The situation we have in Linux is that you – as a proprietary software vendor – not just have to support “Linux”. You need to support a multitude of different configurations and distributions, and this fragmented world is very hard to handle.

First, some clarifications about my point of view on certain “social” aspects in the Linux community, so you can understand me easier:

Regarding FLOSS vs. proprietary software

I think both is important and we should fully support both. For me, I want the OS platform as open as possible to have unlimited possibilities regarding stuff I can do with it. And there should never ever be something limiting my freedom do make any change I want on the Linux platform. BUT, if people want to deploy proprietary software on that free-and-opensource platform, I welcome them too, because everyone should be able to use the FLOSS platform for everything they want. Also, it is better if more people use a free platform and be aware of it – this is the best way to prevent stuff like locked hardware devices and the UEFI hell. More app developers will certainly help in adopting the Linux platform. That’s why I welcome Valve on Linux too, although they’re proprietary and I don’t play games.

I always prefer free software over proprietary software, but there are some valid arguments for developing closed-source stuff too, and I don’t want to exclude these developers and add impose limits for our users to get software.

Regarding Linux and GNU/Linux

What is “Linux”? Many people already did lots of fighting about the name – I don’t want to do this here.

Sure, Linux is a OS kernel – technically. But what is Linux in society? For me, “Linux” is not a kernel or OS, instead it’s a value and brand, which defines all Linux OS’es. When I talk about “Linux” I refer to all components a Linux distribution is made of, the kernel, the plumbing layer, the toolkits, the desktop the GPL and the freedom we want to give to our users, as well as the great community. In population, Linux is a know word for “free operating system”, Linux is recognized – many small distributions are not. If I want to speak about the kernel, I just say “the Kernel” or “the Linux Kernel”.

People port software to “Linux”, people use “Linux” etc. – Linux is used as a brand and word for a free operating system.

So, it is important that software companies can target the Linux market if they want – not only a small fragment of it called “Ubuntu” or “Fedora”, but the whole Linux world using just one software build. People will see a download link for Linux software, not just a simple tarball and recognize Linux as equal (better!) to Windows and Mac.

The Linux world is big – it’s just not recognized as big. The Humble Indie Bundles already showed Linux users pay for software (I never doubted that and it doesn’t really surprise me), so we just need more app developers now.

Mark Shuttleworth recognized the potential Linux has – with programs to promote the Ubuntu desktop, workshops for app developers, the PPA-way to submit software and app-cebtric software management, they’re doing everything right. But of course, they’re Ubuntu-centric, software is developed “for Ubuntu”, you won’t find the word “Linux” anywhere on the Ubuntu homepage. Having a universal software distribution format for Linux would also be fair for smaller distributions, so they can receive new applications very fast too, although they don’t support Ubuntu PPAs.

Listaller changes

Listaller is currently designed to do exactly that – create packages to target the whole Linux “market”. These packages provide FLOSS developers a great way to push new releases directly to their end-users, removing the “distributor” part in the software release pipeline. Proprietary developers instead get a great way to target any Linux distribution.

The package format is based on standards like DOAP, so every application which ships DOAP informations (i.E. every GNOME application) can generate an IPK package really fast.

Of course, there are some limits: Listaller is designed to install applications, and only applications. Installing core infrastructure is not supported. This means you can’t install the full KDE with it or install systemd. You also can’t execute arbitrary code at install-time, most-used tasks are hardcoded.

Removing flexibility means adding security here.

Brought to you by…

The recent changes in Listaller wouldn’t be possible if the Linux desktop isn’t already becoming more app-developer friendly and more consistent: There is an increased awareness of ABI compatibility, systemd is unifying many OS functions which have been completely random before and Freedesktop is a great project which allows collaboration in components which work on all distributions.

For everything not yet covered, Listaller provides tools to make applications binary-compatible with many distributions, but I hope these things won’t be used in future anymore.

The future

There is a lot of new development in the GNOME community, which will make the life of a cross-distro packager and therefore Listaller easier: Stable APIs, a “GNOME SDK” etc. These changes might make it possible to make Listaller packages containing pure GNOME applications just depend on “GNOMESDK>=3.8” instead of handling every single dependency separately.

On the KDE side, the project is splitting into “Workspaces”, “Frameworks” and “Applications”, which is also a great move forward for cross-distro apps and will make packaging KDE apps in general easier.

So, with this new development, I think it will finally be possible to build a cross-distro software installer with a design driven by user’s needs and not creating a big hack and workaround-set to overcome all technical limitations of cross-distro software: We might be able to fix these problems at the source. (The increased caring about ABI stability in FLOSS community already helps a lot)

Also, Listaller itself needs to be future-proof: We have two developments which affect Listaller:

a shifted focus towards WebApps instead of traditional applications

distribution through software-stores

For the first point: I do not believe in WebApps replacing traditional software – there are just too many limitations, and I also don’t like the idea of web apps storing much personal data on servers I don’t control. (Although there are efforts to fix that) I also think the Web technology is in many points inferior to what we use on the desktop today (be it Qt or GTK+). Instead, I think we’ll have more and more “hybrid” applications soon, which e.g. have logic implemented in C and run a UI built with web technology. Also QML is a very innovative and great approach for UI construction.

For the second point: I consider every application store as limitation of user’s freedom, as long as it is the only possible appstore and/or adding other stores is impossible. Competition between appstore vendors is very important.

So I imagine the following (far away) scenario: Distributions ship Listaller with their own software-store enabled, if they have one. The base OS is made of native packages, all additional software (also newer versions) is delivered through Listaller packages. (This doesn’t mean there are no applications like Firefox in the native packages – everything stays as-it-is, Listaller packages are an addition)

Because Listaller packages are distro-independent, OpenSUSE users can also use Ubuntu’s appstore-source, so there is no longer any dominance of a distribution regarding availability of precompiled software.

Ideally, one organisation creates a “Linux AppStore”, which is added to many distributions by default. (but maybe disabled and easy-to-enable) This Software Store will sell applications by commercial software vendors and – because it should be carried by a non-profit-organisation – sends most of the earned money back to distributions, which itself could use it to improve their distribution.

This software store would be very attractive for software vendors, because they could target the whole Linux market and would be very visible to potential customers, on distributions from Gentoo to Debian. Also Linux itself would be much more visible to others.

Listaller packages are by design very secure, so these packages can’t harm the system. (they’re usually signed and don’t install stuff in system directories – also a sandbox interface exists (but is not used at time)) Upgrades will work without problems, because the native package configuration is never touched, so distributors will receive less bug reports about those issues. PPAs will instead be used by people who want the latest Linux infrastructure, like a new KDE, GNOME, systemd, PackageKit etc. version, and not by people who just want the latest GIMP.

Because the setup process is 100% controlled by Listaller (no custom user scripts in packages, many things handled automagically), distributors can control every single 3rd-party software setup on their distribution just by adjusting Listaller to their needs. So distributors still have control about every piece of software installed on their distribution.

Also, Listaller is nicely integrated in all existing application-managers if they use PackageKit, so extra-UIs aren’t needed and users don’t even know that the software is there.

Of course, installing standalone software packages is still possible – they might of course be run in a sandbox by default.

Listaller will have logic to maintain the different software sources and provide information in a way frontends can display it nicely.

Implementation!

Of course I’m working on the code to make it ready for future use. The first step was adding support for package-updates using package deltas in Listaller 0.5.6 – unfortunately this feature has been delayed because of my SoC project and because I completely broke the API Listaller uses to talk to PackageKit during my work on PackageKit. So the next release will be without a major new feature, but with many bugfixes.

To make Listaller releases more frequent and because Listaller depends completely on PackageKit and internal APIs of PackageKit, I’ll sync the Listaller release cycle with the one of PackageKit. This means you will get a new Listaller version one week after a PackageKit release. By doing this I also hope new changes will reach people faster – and there will never be the case where Listaller and PackageKit are incompatible. (At time, Listaller 0.5.5 won’t work with PackageKit 0.8.x series)

Of course, the project needs contributors! 😀 At time, “we” are only me and a few part-time people who create probably one small commit every two months, since the main contributors started to write their bachelor thesis or do other stuff (at work).

I would be very happy about comments regarding the plan above, which is of course very rough at time… To make it more concrete, I’ll try to talk to people at GNOME about their vision, because it seems they also have similar plans or at least have one for application management improvements, which could at least be valuable input.

And finally: Is the Linux desktop dead? The answer is NO! There might have been mistakes in the past, but both KDE and GNOME have clear goals for the future, are more end-user focused than ever, care about ABI stability, collaborate on an increasing amount of projects (although it could be more ^^), prepare for targeting mobile devices and are used by millions of people. We should be very happy 🙂 Let’s see what we achieve in future, even if it is not desktop-dominance over all other OSes, it will be great anyway! 🙂

(Of course I also care about Xfce, LXDE and E17, just before someone asks ;-))

If you want to contact me, you’ll find me on IRC: #PackageKit (Freenode) There’s also a Google Group about Listaller, but I’m looking for a traditional mailinglist at time… Reaching me via mail is always the easiest way 😉

23 Comments

Sounds like a great idea. Chakra has their own way of doing these types of packages (called “bundles”), so why not go to their dev mailing list and try to pitch this to them? It’s a small distro that makes up a lot of their own infrastructure (since Chakra is no longer based on Arch). I’m sure Listaller can take a lot of the burden off the distro’s maintainers for their click and run packages! It’s just a suggestion.

Chakra and the bundles currently do not depend on packagekit, due to a lack of implementation of alpm/pacman in packagekit. Furthermore, Chakra is intending to switch the package management from pacman to akabei (29th of august the alpha1 was released of akabei), which is almost finished. Akabei probably never ever will have a packagekit backend, since it depends on Qt and libalpm currently does not have any apis wich do not depend on qt.

Ye, I know there is actually an implementation, which is neither complete, nor it works properly, especially with newer alpm versions.

The worst part is, alpm is almost undocumented, and hard to be implemented (at least in the state 2 years ago).

Yeah, I can see the reason why Chakra devs made their own package manager. I didn’t knwo the alpha was just released, I’ll go take a look at it (though I did install Akabei from git before but I didn’t make my own packages using it).

STIAT already summarized what I wanted to say nicely 🙂 A dependency on Qt is no problem for PackageKit, backends can link against whatever they want, so Qt really is no problem – we already build packagekit-qt from PackageKit source 😉
I’ll add the Chakra Bundles to my watchlist (didn’t know they had this! thanks for the hint!), but I am quite against the bundle-idea, because it creates additional problems which make managing & updating software more difficult, as well as sharing resources.
(But regardless of that, interesting to know – I very much appreciate the work of the Chakra developers, and they’re open for new ideas)

When Chakra adds PackageKit, Listaller will be available too – and it is in theory possible to implement Listaller support directly in a package-manager, avoiding PackageKit. But this is not worth the effort, having a great PackageKit backend for $PACKAGE_MANAGER is definitely better.
And btw, there have been some recent changes on the alpm backend of PackageKit…

Sounds promising, but I still got a few thoughts about that. Need to look at the implementation for that, but ABI compatibility and similar are quite a hard-breaker nowdays (boost,OpenSSL, ICU and others .. as we learned lately).

If you solve that by installing all depends in a sandbox, it blows up both, ram usage and loading times, you get rid of all the benefits the linux kernel filesystem cache provides. It solves problems, but could cause others.
Sounds pretty much like the bundle systems Chakra has got. Though – no details about implementation there.

Worth a look in any case. And a chance too, I’m sure about that. The wiki didn’t really give enough details to get a proper opinion.

Thank you for this great plan, but I think it is very, very hard, I like most of what you say but have a different opinion about AppStores, I think they are very important and of every distro.
App developers must help you too by not relying on architecture specific apps (I was trying to install PeaZip on Mageia x86_64 today with no success because it uses some 32-bit-only code).
I have some opinions and hope you consider them:
1) This vision is great and very important and will have big influence if completed, so please, make it happen.
2) ABI changes are our biggest problem that needs to be solved; however, Linux Standard Base (I know Debian people hate it) is a good start, when you want to install Google Chrome, it needs LSB 4.x, it ensures that minimum required software is installed (e.g. libc, Linux kernel, etc..).
3) Updates are very important to support (optionally), with upgrades (even those that needs money) available to companies that want it, and about binary difference, there are many ways, Fedora way which is version sensative, openSUSE way which is great, Pardus way that I don’t know how it works, I hope you can pick the best way.
4) A great example to look at is PC-BSD bundle system, sandboxed bundles with no duplication (through the use of hash directory) and support binary diff, please take a look at it:http://blog.pcbsd.org/2011/04/package-management-for-the-upcoming-pc-bsd-9/http://2011.eurobsdcon.org/papers/moore/eurobsdcon2011-pbi9-kpm.pdf
really hope you read it, it has great ideas.
5) We need two ways to install software, the first is installing from the Software Center (with AppStream) and we can also use web interface like openSUSE 1-click-installers or Android apps, and the second is by going to the Developers website (e.g. http://www.firefox.com) and grab a standalone package to install it anywhere (even offline), and the package must be able (optionally) to add a repo to the Software Center (like what Google packages now do) for updates.
6) Installing multiple version of the software must be possible, especially if they are in separated directories (PC-BSD way).

Sounds super exciting!! Thanks for the post and best of luck in implementing this great vision 🙂 It would be cool if the linux foundation were to host a pan-linux migration friendly app store using your suggestions/technology

Listaller differs greatly with Muon, Listaller is is distro-neutral installer of 3rd party apps that runs on almost every distro, Muon is Debian specific (it uses QApt not PackageKit) and used mainly for installing apps from official repos.

A comparison like this won’t help much, as Muon is a native package manager, and Listaller is a completely different concept.
Also, Listaller does not have an own UI, it relies on PackageKit frontends like GNOME-PackageKit and Apper to display it’s stuff.

The more interesting way to go is this one: Integrate Listaller into Muon-Discover and make this part distro-neutral using PackageKit 😉
That would work very well, but requires manpower.

Does anyone know why kubuntu is focusing on Muon instead of PackageKit? It does seem a little weird to be turning back from a bigger project like PackageKit – which presumably will have a tonne of great upstream development – to a distro specific homespun solution like muon??

PackageKit had a few problems when this decision was made. Also, Muon offers some powerful Apt-functions for advanced users, which Apper can’t. (and which most users won’t use anyway)
Apper has IMHO a much better UI (it just lacks the application-center part).
The development power of Muon is bigger than the one behind Apper, it’s basically just Daniel and me making changes. Compared to PackageKit, Muon would loose regarding this small aspect, but PackageKit is just the non-visual part of a package manager.

Since you mentioned ABI stability I guess you are at least partly referring to Miguel de Icaza’s lates troll blog.

Any serious library project cares a lot about ABI stability, especially those providing a whole set of them, e.g. GNOME and KDE.

Sure, some smaller teams doing some utility libraries might not have the same level of understanding yet, but those libraries can easily be shipped with an applications, just like on any other platform.

You sure about that? Even “older” and “taller” projects don’t (counting OpenSSL, ICU, boost and a lot of others if you think on what needs to be repackaged even on minor releases).
I’m just taking the ones out of Thiagos post, there are [a lot] more than those three having problems with binary compatibility even on minor releases.

Well, boost is mainly templates and it is one of those one would usually ship as part of the application.

OpenSSL is known to not have any kind of stability which is why basically every application dlopen it instead of linking it at build time.

No idea about ICU.

For me they all fall into the category of utlility libs that whose authors just want to provide functionality and do not expect to be a platform API. This kinds of libs are just bundled with the application on all platforms.

Some of the libs, as mentioned by poster before, don’t care that much about ABI stability.
But in these cases Listaller can try to fetch the dependency from elsewhere. Also, developers can use the LiBuild facilities to link their software against older library versions so it will run on older distributions too (and the native packages satisfy the dependencies).

And yes, this was for Miguel 😉 I usually disagree with nearly everything he says, but this time he has found a point. He hasn’t found the reason “why the Linux desktop failed”, I don’t think it was because of ABI instability. But at least he has a valid argument 😉

(I just wanted to stay a little bit outside the big fight on Google+ about the Linux desktop, that’s why I avoided quoting people above)

I’m sorry if i failed to read the pertinent section of your post, but i can’t find where you’re installing the packages.

I’ve thought about an external package manager for all distro’s before, but i’m not in IT, and only have some clues in Python.

I look at Firefox, which i can just untar in /opt and run it from there, and a few other apps/games that work the same way. All these programs want is write access for /home to write configuration files, and so on.
I can’t help but think that the solution is right here. The only problem is updating the applications, which is a manual chore.

Why not simply making a program that untar/zips a file to /opt/programname.

The key features would be that it would install in /opt only, check and update the programs, and new people don’t need to use the command line, or some elevated file manager, and you could provide a default central repository where application developers can publish their programs with pretty much the same type of community + host oversight as in addons.mozilla.org.

One could browse via the web page where the install button opens the native program for download and install, or browse via the program itself like Synaptic. Packages would obviously be signed by the repo.

Perhaps the program could support distro dependencies.
For instance, i package Firefox – package.extension would include firefox.tar.gz and a text file with dependency resolution details. I tested Firefox on Debian and make note of the Debian specific packages required, and put the in the text file.
When installing on a Debian system, the package manager would check the dependencies and install them, marked as such (so that uninstall would clean the system).

For distros not tested, the package manager would complain that it was not tested for that distro, and suggest either not install it, or proceed anyway (after all, all it really needs is to drop it in opt). The user could still try the app, and if he finds that it has extra dependencies, he could satisfy them himself and help the repo with the info, or uninstall it.
If it simply does not run regardless of dependencies, report as broken on Distro X version n.

After that, one could think of how to sandbox these programs for good measure.

What kind of programs wouldn’t install this way, and why? What kind of problems could arise? I’d really like to understand this better.

Wouldn’t this be simple for application developers, and users, clarifying and separating two different systems?

What you describe is btw exactly what Listaller does and partly what it has done before the rewrite.
A Listaller package is basically a compressed tarball+signature+software-description and other metadata+dependency information+install-recipe.
The install-recipe tells Listaller where to put the files, default points to /opt/appdata. (If packages install in unsupported directories, users will see a security warning)
Tarballs are difficult do handle for non-technical users, also the tarball content is not registered in the system (desktop-menu! package-manager!) and – you said it – difficult to update. Listaller takes care of this stuff.
The web-store you describe is described by me in the post above 😉

Pre-rewrite versions of Listaller allowed packagers to define a “recipe” for every distribution they wanted to support explicitly, which contained a package-list. This feature has been dropped, because it really was an anti-feature which was not compliant with Listaller’s goals and it was something which belonged to the distribution native tools.
Now Listaller uses PackageKit logic to fetch dependencies, if the distribution provides appropriate metadata. (which it should do!)

Because Listaller controls 3rd-party installations, sandboxing is really easy, it can be added with only a few lines of code.

For users, separating two systems which both cover “application-management” is no good idea – it is highly confusing and users will never know which tool they should use. One GUI interface for all is best. Also, you don’t want to run two software-update services, right? 😉
Listaller provides all the automatics to make 3rd-party application installations easy & intergrated with existing tools.

For Listaller design I use these profiles as a guide: http://www.packagekit.org/pk-profiles.html
I also take a look at developers and distributors needs when adding new features / discussing about suggested features. It is very important to find a balance here.