Introducing QApt and the Muon Package Manager

It has been a while since I’ve blogged. I know I said I would post stuff about something exciting at UDS, but that sort of never happened. 😦

But! Better late than never. I think that waiting a month or so has also made this post better than it would have been at UDS. Before I do any introducing, I feel that a summary of the history of KDE/Debian package management is in order.

A Brief History of Debian-based Package Management in KDE

It can be said that the hardest part of writing a package manager is not in writing the GUI, but in interfacing with the package system. For a long time, this has been the biggest barrier that has limited the number of Qt/KDE package manager for Debian-based systems. Even among those that do exist, such as Adept and PackageKit, each interfaces with your computer’s underlying package system differently, each with its own set of bugs and unique, not-always-equal feature set.

These factors combined have caused a drought in the number of Qt/KDE packaging systems from the earliest days. When these package managers were good (like Adept 2 in KDE3 days), things were generally fine, and the drought had a limited effect. The transition to KDE4 threw package management GUIs right back to square one, and even two years later there has not been a replacement of the caliber of the KDE3 package managers.

But what if there were a way to alleviate both of the factors contributing to the lack of stellar package managers? What if all Qt package managers for Debianish systems could share one underlying library to interface with the system package cache? What would happen if the success of a package manager was not limited by the difficulty of writing a low-level package management backend, but rather was solely dependent on the quality of the user interface?

Introducing QApt

QApt is the answer to the above question. I had pondered the above questions for quite a while, starting back with the inception of the Project Timelord initiative.

A bit later, the Shaman project was announced, and I was interesting in writing a backend for Shaman. There was already an APT backend for Shaman, though it was in its infancy. That’s when I realized that libapt-pkg was an ancient dinosaur with an awful API. That’s when it dawned on me that I could write a library/backend that any Qt or KDE application could use to manipulate/read the Debian package cache. If a focus on this potential library could be made, then any Qt or KDE application could use it, from Shaman to a “Software Center” style app to a completely custom, fully-featured package manager. Developers would be free to focus on making a great GUI, and could all share the same internals.

I started three months ago by spending a few days trying to grok the libapt-pkg API by looking at the internals of Synaptic, the APTcc PackageKit backend, and the python-apt bindings. After a month or so of hacking, the result was QApt. Then came finals season and UDS…

But what is QApt? QApt is actually split into several parts. Half of QApt is LibQApt, which is a Qt wrapper around libapt-pkg that boasts a saner API, as well as a pretty extensive APT cache implementation for fetching information about packages. This constitutes the read-only portion of QApt. The second half, the QAptWorker, is equally important. The QAptWorker is what does all the dirty “needs admin” stuff, and could be considered the “write-only” portion of QApt. It handles everything from checking for updates, doing the package downloads, and actually installing/removing packages. LibQApt interfaces with the QAptWorker over Polkit, giving all the normal security and sysadmin-flexibility of the PolicyKit framework. Apps using LibQApt do not have to run as root, and sysadmins can exercise fine-grained control over which operations they let their users perform.

In addition, QApt ships with a simple batch installer utility called “qapt-batch”. It is a drop-in replacement for install-package, which is a utility that several Kubuntu applications use to install packages from non-packagemanager applications. (E.g. codec install, the firefox installer, apturl-kde, etc) It already has a few neat features that install-package does not, including more informative error handling, security improvements (warn about unsigned packages, etc) and media change support.

For further technical details, you can see the slides from the UDS presentation on LibQApt that I did here, and the neat little diagram I made with umbrello here.

What uses QApt?

At the moment, two external projects use it. There is an infantile Shaman backend based on QApt in the shaman svn branch that can do your basic things (install, remove, check for updates, error handling, etc). Check out playground/libs/libqapt, install, and you should be able to build the shaman backend.

In addition, I wrote my own package manager, Muon, which I shall spend the rest of this post discussing.

Introducing Muon

Muon is a package manager based on QApt that I started to write about two weeks ago, when I felt that LibQApt was extensive enough to be able to write a package manager with. It follows in the current KDE naming fad of naming your app after a subatomic particle. The purpose of Muon is to cater to the usecase ranging from “user who knows a fair bit about how his computer works but really isn’t a terminal-lovin’ power user” to the “poweruser” usecase. Its goal is to have all the power of the end-all of all Debian-based package managers, Synaptic, while at the same time presenting this functionality in a more usable fashion than Synaptic. It’s never going to be a “Software Center” in terms of “Your non-techie family member could use it”, but that’s OK; thanks to QApt, writing a Software Center-style app would be just a matter of writing another GUI shell. Seeing the amount of progress I made in two weeks, this is definitely not something to worry about. (I also plan to write a Software Center/Adept Installer style app once Muon has a stable release)

Power management suspension during package downloads, installations and removals

Support for download the latest changelog of a package

Package screenshots

But, talk is cheap. Bring on the screenshot tour!

Here we see the initial view of Muon. (Click all screenies to enlarge.) Pretty straightforward. Toolbar on the top, filters to the left, search bar/package view to the right.

Here I’ve searched for Konversation and clicked the first result. A details tab bar popped up as a result. The first tab houses the short description, a button to fetch a screenshot, buttons for marking konversation for removal, purge, or reinstallation, the long description, and finally a short blurb telling how much longer Canonical will support konversation. (Since it’s in main)

I couldn’t fit all the details in the first tab, but the second tab has a bunch of neat technical details.

Here we can see Konversation’s dependencies. For packages that provide virtual packages, you can also view those packages by selecting “Virtual packages provided” from the combobox. But Konversation doesn’t provide any virtual packages, so this screenshot can’t show you.

The changelog tab. (Hey, that’s me! :D)

Since I already have Konversation installed, I can see what files the package has.

Filterting by category works.

You can filter by category and by status at the same time. In this picture, I am filtering on installed KDE packages.

After I marked amor for installation, I hit the “preview changes” button in the toolbar, which graciously shows all packages to be installed/removed, and turns the preview button into a “back” button.

Gotta give my password before I can hit apply. (Polkit remembers the password for a few minutes, so you can install/remove something again directly after this install without being annoyed by the dialog again.)

Downloading. This could stand some improvement, (Per item progress, get rid of the URIs, etc) but is functional.

Installing amor. It happened so fast that it was already cleaning up by the time ksnapshot could get a picture. 😉

If the need ever arises, a debconf GUI shows up. Please don’t mind the Debian branding. That should change before any stable release. Muon uses the Debconf KDE Library written by dantti, which should hopefully also be usable in the next version of KPackageKit. 🙂

Untrusted packages, while usually just a matter of forgetting to add a GPG key for a PPA, are a security concern. (E.g. DNS Cache poisoning attacks…) It does follow APT configuration, though, and will totally disallow installing unsigned packages if that is set in /etc/apt/

Muon also asks if you really want to remove that essential package

..and also prevents you from making changes that would break the package cache, like any good package manager should. (It undoes the attempted change after you press OK)

For all its features, Muon and QApt are not yet, in my opinion, ready. They still need support for showing reverse-depends, package downgrade and pinning support, and in general just need some more polishing and bughunting. Muon still is only two weeks old, you know. For this reason, I’m going to say right now that this will not be the default package manager for Kubuntu 10.10. Every single time we’ve jumped on the latest-n’-greatest piece of package management shininess, it’s never ended well. Adept 3 was decent in my opinion, aside from search accuracy and a somewhat scattered UI, but it was rushed to be made ready for release and really never stood up to the quality that Adept2 provided. Similarly, K/PackageKit was still young when we picked it up in Kubuntu 9.04, and while in 10.04 it’s not the disaster it was in 9.04, it’s still not as nice as a package manager could be. So let’s just let Muon and QApt cook for a while, and then we can see where we stand for Kubuntu 11.04. It’ll be worth the wait. 🙂

In the meantime, I plan to go through a prerelease cycle for both QApt and Muon. It is conceivable that at the least qapt-batch could be made ready to replace the python install-package for Kubuntu 10.10, but aside from that nothing big will come. I do plan to make packages for both QApt and Muon available for lucid and maverick throughout the prerelease cycle.

You can grab Kubuntu packages of the first alphas of both Muon and QApt from my PPA for both lucid and maverick here. Muon should also build and work just fine on Debian, but I don’t have packages for it. It requires Qt 4. 6 and up, so users of Debian unstable should be able to build and use it from svn just fine. (muon is in /playground/sysadmin/muon) Tarballs for QApt 0.2 (1.0 alpha1) and Muon 0.2 (1.0 alpha1) can be found here and here respectively.

Please beware that while things should in theory be safe, Muon has not yet had widespread testing, so there is still the slight chance that a bug could still be lingering, ready to kill your kittens. Just keep that in mind, and have fun. 🙂 I look forward to feedback on Muon, and will continue to work on improving both Muon and QApt for a brighter future for package management.

Post navigation

73 Responses to Introducing QApt and the Muon Package Manager

Congratulations, thank you (and Dantti) for bringing good software handling to KDE. By the way, I do not really understand Launchpad. I have a Launchpad name but do I have to be in some sort of “team” to report a bug in some random third-party PPA? I will try this somewhere today, but it is late now. If I wanted to, and it would be okay with you, where do I report problems?

Hmm, bug reporting seems to be something I didn’t entirely think through… Since the code is hosted at KDE, I’d like to use KDE facilities for such things. For lack of a better method, feel free to post them here in the comments, I suppose. I’ll try to get a bugs.kde.org bug product for both muon and libqapt tomorrow, as it’s quite late now.

Awesome job, keep up the great work. In september, as soon as my free time is back, Shaman’s internals will be finalized, and I’m looking forward to port your frontend to libshaman, as I really like its design 🙂 It would be cool to have it as a “generic software center”

Looks like nice work.
Sorry to make a ‘my 2 cent’ post, but I felt compelled to:
I never could adapt to adept for some reason, either version.
I disliked kpackagekit, and stopped using entirely, after I tried to install something, and got an obscure and meaningless error message, and when I tried it with synaptic.
Synaptic failed too, but it let’s you see the output from dpkg. Even without being a super admin, I was able to fix the error.
The point is that every package manager should have an option to show the dpkg output. if the idea is not to scare the simple user, it can always be hidden like in synaptic.

Synaptic’s quick search function, failed me sometimes in the past, even on exact name matches. This could either be due to a bad search algorithm, or the search cache not updating properly.
I stopped encountering it since lucid, but I’m not searching packages that often. in any case, you should look out for that.

Showing the actual dpkg process as it would appear in the terminal is some pretty deep black magic that I haven’t quite figured out yet. It is on the TODO, assuming I can figure out how to do it. Getting those few lines of text that do appear is a little bit of black magic in itself…

On the synaptic search front, I ported it from the latest version in maverick, which contains a few bugfixes over the search algo in lucid. 🙂

Very pleasing to see that someone has token the very important tasks to their tasks list what will improve a lot the KDE SC use among normal users!

What suprise me is that dpkg is so difficult package system to manage. I have always tought that Debian makes very clean and simple code!

But doesn’t this use APT (package manager) at all then but just dpkg (package system)?
So how well it actually can resolve depencies etc? APT is great doing that (nothing else I can not find from it what others is not doing. My opinion is that APT is already too slow and difficult even from CLI state.

Hopefully usability experts can step in and check then all those GUI’s developers/distributors start making…

Looks great! I’ve been recommending opensuse for people that ask me about KDE, in part because I think the package management hasn’t been that great in the latest kubuntu releases, so kudos for improving it.

Some suggestions:
– Why not try to automatically get the screenshot (in an async way, of course)? You could show a little square with a spinning-style cursor thingy inside (like rekonq does while loading a page thumbnail http://picasaweb.google.com/adjam.photos/Rekonq#5390173251261664626 ), and if there’s no screenshot, just replace the spinny thingy with a “No screenshot available” message

– Composing multiple types of filtering + search seems to me that could confuse users: they might choose a category, then switch over to status and “forget” they chose the category. May I suggest adding something like the dolphin bar to help the users? Then you could display
Showing: KDE Software > Installed > Named “konversation”

– During operations, why not hide the huge window and instead use a smaller synaptic-like window. You could even display the screenshot, name and description of the apps being installed, so the users could read along if they wanted.

– When displaying the installed files, may I suggest not displaying intermediate paths: in your screenshot it shows
/usr
/usr/share
/usr/share/doc
until we finally get to what’s important /usr/share/doc/konversation
and also maybe the paths (or even the files?) could be clickable, so that you can easily open the docs directory by clicking on it.

– Maybe stuff on the dependencies list could also be clickable (would get you to the referenced package in a aptitude-like way)

– Also you could capitalize the first letter of the small description text, for example for konversation instead of displaying “user friendly Internet …” you could show “User friendly Internet…”

– Could the package icons be derived from their category? The KDE category could have a little K icon that KDE packages would share, same for gnome, etc… and that way you could quickly visually determine if a package “”””belongs”””” to kde or gnome.

Sorry for the laundry list of ideas, but I really like the screenshots, it seems to me you could very close to have something with the power of synaptic but that is more usable and user-friendly (and that looks much nicer than opensuse’s yast package manager).

Regarding automatically getting screenshots. I’m not sure that this is a good idea. We have to keep in mind that at least part of the user population is using metered-bandwidth mobile internet connections, where they have a limited amount of bandwidth they can use, and anything over costs a lot of extra money. I do wonder if there is a way to automatically at least “ping” screenshots.debian.net for the screenshot using KIO. Will have to look in to that.

As for filtering, I’m not sure that the target audience would be confused too much by this. I think I’ll put that one on a more “wait and see” basis. But thanks for the input. 🙂

For the “committing changes” view, the reason it is in the main application window is that I don’t really like popups all that much. If/when there is an optional konsole view of the dpkg process, this might become less of an issue, but for now I don’t think I’ll go the external-dialog approach for the commit window.

I actually hadn’t noticed that the intermediary file paths for installed files were showing up. It’s unfortunate that APT hands us the file list that way, but I did fix LibQApt this morning to not show thoses. 🙂

Clickable dependencies would be neat. I suppose that would require a linking system of some sort, but it shouldn’t be too hard to implement. Some problems I could forsee with that is that the new package might not be in the user’s view due to search or filter parameters. Would have to think about what to do in that case.

The capitalization of the first words of short descriptions are unfortunately out of my control, as these are defined in the packages themselves. I have always wondered why this is preferred packaging policy to have the short description start off with a lower case letter… oh well.

Thanks for all the feedback. It led to at least one improvement this very morning. 🙂

You’re right about limited mobile internet connections, but could this be added as an option that was on by default? That way, bandwidth-conscious users could turn it off. I really think this feature would benefit the user experience a lot.

Seems like one I have been waiting for.I’ll say the opinion after using it .Where can I submit the bug reports ?
Iam curious to know whether you will actively maintain(or plan to maintain) the softwares .

Please report bugs to the “Muon” product at bugs.kde.org. Since the alpha was released before muon was in the bugs.kde.org system, you can’t report the bug from the application. You’ll have to report it manually until alpha 2. Sorry about that. 😦

Thanks for that, just installed it and am please that there is finally something a the level of synaptic for Kubuntu. The only small problem is that installing the muon package doesn’t pull in libdebconf-kde0.

I would suspect that this is a problem with libdebconf-kde itself. It doesn’t seem to be installing a .so versioned library, so it doesn’t get picked up as a dependency in the package building process. (The debconf lib is young, too.)

Very well done… great work! 🙂 And I apologize for the stalled Shaman development atm… we’re all running out of time and without a good and stable backend for ArchLinux it’s not really possible for me to fix it atm

A feature that I’d like to have and that I have never seen is a more flexible installation of the packages. When installing some packages, I’d like the software manager to be like a download manager:
Being able to add a new package to the queue after the installation began, deciding that you want to move up a package and move down another, pausing the current installation,…

The typical use case is when you first install your distro, you update to the last version of KDE SC and you have 2GB of packages to download. You start it and then you realize that you want to look at a video and don’t have the codecs installed. And either they are at the end of the queue or not in it at all.

Of course I understand there will be some issues with dependences, but I think there must be a way to overcome them. I know that is a lot of work for a so little benefit, but I’m sure I’m not the only one realizing I wanted to add other packages after having started the installation, and then forgetting it.

I’m not quite sure that this is within the abilities of APT itself, QApt regardless. If it were, it would be no small undertaking. I should note that as long as the download hasn’t finished, you should be able to cancel, add your new package, and re-start the process, and it will pick off where you left off. (It saves the files its downloaded so far)

Awesome work,from the past 30 mins of testing,muon hasnt crashed once,even grub updates seem to have gone correctly ( will have to reboot to check if muon hasnt eaten grub ),but apart from that i didnt find any serious flaws in the package manager.For a alpha release it has all the basic functionality one would expect from a package manager.
A +1 from me 😉

I installed Kubuntu Maverick as a guest OS in VB. After all the usual updates I went to your ppa page and updated sources.list using the command line. Then I issued
sudo apt-get update
sudo apt-get install muon

It installed without any complaints, but did NOT put an option in the kmenu (which may be my issue). Anyway, I hit Alt-F2 and in the command line box entered: sudo muon

It came up without any problems. I played with it for a while and noticed an item that needs addressing:
1) I could not find a menu option add or remove a repository, or to reload one if I could.

and I have a wish:

2) The BIG weakness in Synaptic (and KPackageKit) is that only the currently installed driver or the next version of it is listed for installation or removal. IF the next version is installed all references to the previous version are lost, which makes it impossible (via a gui) to REVERT to the previous version if the new version doesn’t work out. Ergo, it would be nice that when a package is upgraded the previous package reference is stored to allow a quick reversion to it if the update is not satisfactory.

After installing any KDE app, you have to run kbuildsycoca4 or log out/back in before it will show up. A bit annoying, yes, but nothing I can do about it from my end.

1) Yeah, I plan to make a menu entry to launch good ol’ software-properties-kde in there soon enough. (It’s the repository manager that both Adept and KPK use in Kubuntu) It’d be nice to have a custom solution for that, but that will probably take time. To reload existing repositories, simply check for updates. (That does the same thing.)

2) Actually, Synaptic does have a way to downgrade package versions. It’s just not completely obvious how to do that from a cursory view at its UI. You have to go to the “Package” menu, and if there is an alternate version still available, you’ll be able to downgrade it. This is something still on the todo list for LibQApt, but it is definitely planned to happen.

Great work so far, this looks simpler to understand and use than KPackage Kit. Looking forward to the eventual production release.

One thing that’s always bugged me about installations is that when you type “Konversation”, for example, in your search criteria you get back both the application along with other related packages (e.g. “konversation-data”). It would be really awesome if there was a way to filter out supporting packages in the default view so that in general you only see a list of apps instead. I think this would make newer users’ lives much simpler in terms of managing their software. Is such a thing possible or are there technical issues with being able to properly determine this sort of thing?

This seems to fall more within the realm of a Software Center or Adept Installer type of application. Both of these use the app-install-data package to provide information on which packages are actually applications, and provide interfaces tailored towards installing applications rather than packages. (You can see Adept’s app install interface by installing adept, and then running it as “adept installer” rather than just “adept”.

There is room for both types of tools, and I don’t think that there is any way to combine them in a satisfactory manner. Sometimes you need a tool for managing packages, and sometimes you just need an easy to use interface for installing or removing applications.

Anyhow, I do plan on making an application installer shell, just not at the moment.. (As I detailed in my post) 🙂

Those really are some sweets screenies. I’ve always disliked KPackageKit and don’t want to pollute my system with ugly GTK stuff just for Synaptic (which inherits GTK’s ugliness and wastage of screen space *g*). So I settled with aptitude, which does a great job. But I’ll definitely give Muon a chance in the future (I’m still quite new to Debian, coming from Gentoo, so I’m still a little cautious).

But now to my real feedback:
– the trend goes towards widescreens, and though I’m blessed with a mid-res 4/3, I’m always thinking of the people with widescreens (both lo- and hi-res): what do you think of the possibility to have all three panels side by side? (Category list, package list and details panel) That would mean less scrolling and more effective use of space.
– despite the package icon and alternating background colour in the package list, lots of similar-looking text can become unclear. What is your opinion on writing the package name bold?
– Actually, one of the first question in my mind while reading your article was “Now, how can I see all installed KDE packages?” Though the current filtering is quite logical, the UI needs improvement. I’m not sure how, though. The only thing I can say about it is that the fact that only one panel can be viewed at one time suggests that they also preclude each other’s functionality. Hm… the status filter list is not very high, so perhaps it can be shown permanently.

Many times installing a package results in additional package being installed if one doesn’t use the CLI.

A menu option to access a history of installations and removals tracked by date would be a nice feature. Using Synaptic I have found it very handy, when backing out a package installation, that I could refer to the history to identify dependencies and other files that were added during the install but would not have been remove by merely uninstalling the original package.

I tried it. Wow amazing . Really excellent work !!!. Considering that you took only 2 weeks to code it, it’s really great .
Muon works very well. It is cleaner than synaptic,and is far better than kpackagekit . It does whatever it can do ,well. Great work!!

Can I request a feature? It would be better if there is a way to know the total amount of data to be downloaded ,for installing . Since there would be multiple dependencies, the size shown on package will not be the data downloaded.As you know,a lot of people don’t have unlimited net,as well as limits are there for monthly usage . It is a must have feature for such people
The data size can be shown either in a box near “apply” Or when one click on “preview”. It’s your decision

I should clarify that it’s just the GUI that took two weeks to code. The guts took a month or two at the least. 🙂
I do plan to get the total download size info in, but I have to do some coding in LibQApt before it will work. Once that’s done, the total download and install sizes will show up in the status bar beside the rest of the info.

Suppose we install package kubuntu-desktop . along with that a lot of packages will be installed . But removing kubuntu-desktop will not remove those packages . Can you put an option to do that? Else, after a few installs and uninstalls (only graphical) there will be a lot of un needed packages installed.

It would be better if there is an option to remove packages installed by a package ,say “x” , even after removing “x”.

[…] and Muon. (For an intro to those who may be seeing this for the first time, read my previous blog post) Packages will be in the same location as before, but are not yet ready. It seems that the PPA […]

I wish you the best I wish you the best and I hope that finally we will have got a good package manager.

On the other hand, If you will use Rekonq instead Firefox in Kubuntu you will have to keep it updated with each release and not have to wait for the next Kubuntu release, which is also needed for K3b (2.0 is in the street and Kubuntu comes with unstable versions).

Nice to be free to scratch in itch in the classic tradition of Linux development, isn’t it? Everything get redeveloped with the latest tools for the latest hardware. If it doesn’t, it usually dies quietly from lack of use.

Muon is an excellent package manager built with Qt4. As such, it does what KPackageKit hasn’t been able to do yet … run reliably. And, unlike KPackageKit, its GUI is more like Synaptic, which is easier to use than KPackageKit.

On Synaptic, you can select programs by clicking on the box next to the program, and installing or removing mass amounts of junk programs that one does not need with a few clicks, On this muon program you have to Right CLICK, Then click install or remove? That\’s double the clicks!

Also it would be nice if you could have the option of listing download size and installed size Synaptic also does this, Its useful when your building your own system with refracta to keep the ISO CD size..

Qapt seems to be buggy, When i installed refractasnapshot-gui, Qapt also installed, Gnome keying, and some other none needed junk, I don\’t use qapt much, so i dunno if does this with other programs