Details

The problem:
The current method of upgrading kernels will remove the current kernel, modules, and initrd on upgrades.
This is a problem because: imagine if the new kernel has a serious regression in it on your hardware, or worse yet, it doesn’t boot at all. Now all of a sudden, the user is stuck with no clear, straight forward path to fix the issue.

In a nutshell:
kernel26 will become a dummy package with a dependency on kernel-${version} and symbolic links will be updated post-upgrade to the newly installed kernel/initrd.

Detailed reasoning for this solution:
Every kernel PKGBUILD will have a version number in the pkgname. For instance: instead of it being just kernel26, it’ll now be kernel-2.6.30. kernel26 will now be basically a dummy package that only depends on kernel-2.6.30. When the maintainers go to update the kernel, they'll update the kernel package, and kernel26 package with the new appropriate kernel version number (this will be what? Maybe once a month or so.)

Now, vmlinuz26, kernel26.img will need to obviously be changed. One option would be to automatically upgrade grub/lilo/etc with an appropriate rule. I don't think most people who use Arch would like this option terribly much. So another option would be to have symbolic links in /boot for vmlinuz26->vmlinuz-2.6.30 changed to vmlinuz26->vmlinuz-2.6.31 upon upgrade. This way, the old kernel is still easily accessible on upgrades, and upgrades will still be seamless (and more or less, act like they do now.)

Possible problem:
One problem with this approach would be users will be responsible for periodically removing old kernels. For this, I don't know of a simple solution.

I don't think this one small issue should stop the idea from being shot down entirely.

It *does* add maintenance costs - on the dev side: changing the kernel26 dep and renaming the svn dir for each 2.6.X rebuild; and on the user side: manual maintenance in removing the old package when satisfied and changing grub lines.

Additional thought: the kernel26 package can contain dummy symlinks to files in the current kernel package, this way allowing grub and things of the like to remain unchanged for the purpose of tracking the most recent kernel.

This isn't really about kernel stability. It's more about the fact that if I upgrade my kernel, I can no longer plug in usb devices until I reboot, which is a tad goofy.

Additionally, lts is a different kernel. This mechanism allows for much easier and clearer upgrade paths and testing of new kernels, whereas the lts kernel follows a different path, and releases less often

lts is way too old, it is not even guaranteed to work right with Arch lon-term (for example, udev will soon break).

I like the idea of having some fallback kernel, and this has been suggested, discussed and dismissed a lot. The problem is that pacman doesn't provide a mechanism to do this cleanly, and the solution suggested will result in a number of small issues we will have to work around, making everything quite complex. In short: I don't like it. I am hesitant to do anything like it without proper support for it in pacman.

Do you have an alternate suggestion? I think this one is pretty good, and has a small number of problems which are fairly minor in the grand scheme of things. I'm interested to hear if Dan has any input

Yes, I hit the module loading problem a lot (it really sucks when you need a VPN connection and the tun driver is no longer available). I would love to have some solution to this whole thing.

So a list of things to consider:
* We currently hack the .Y (2.6.X.Y) version out of the kernel; if we do versioned installs do we continue to do this? I don't see how we can. It already seemed a bit odd to me, but I understand the intent (to not have to rebuild external kernel modules). How many of those do we have anymore?
* kernel26 being symlinks makes some sense. How would we pull in the new kernel packages? Bump kernel26 each time to depend on a newer version of the numbered packages?
* It would be trivial, but a cleanup script provided by kernel26 to only keep the last few versions would be nice. We don't see our users as stupid, they can do this as they please. Those with constrained /boot/ partitions will need to be careful, especially since pacman doesn't handle full partitions that well...
* What support from pacman would help here? A concept of a multi-versioned package or something? I can see other cases where this would be helpful (python 2.4, 2.5, etc.) but I'm not sure where we need some extra logic in the package manager. Another meta version parameter?

@Dan: The idea is that package kernel26 depends=(kernel-2.6.30) [or linux-2.6.30]. On an kernel bump, the kernel-2.6.31 package is added to the repos (svn cp or svn mv) and built. The kernel26 package is bumped to depend on kernel-2.6.31 (most likely when moved to core only). This way, an -Syu will find a kernel26 upgrade, and download the new dependency.

I think the kernel version hackery should stay, and is a separate, unrelated issue. That is more about the EFFORT to rebuild everything, not the impact on the end user.

On a side note, our kernel26 PKGBUILD still scares me, it seems overly complex. So instead of beating around the bush, how about something like the attached? Diffing with the current kernel26 PKGBUILD is probably useful as I just hacked that.

Only problem is we slightly lose with the current makepkg, as it doesn't deal with the way we need to do dynamic things very well (note that the function it will end up calling is an alias as bash doesn't support a declare syntax for dynamic function names):

Thank you, Harley. Its been almost like the emperor's new clothes. Everybody seems to want to ignore the obvious. That this is necessary goes without saying. Imagine how many people are dead in the water after an ill-conceived kernel change. Yes, the only issue is how to implement it.

In this unstable environment, nothing should ever be forever unless the user wishes to deleted the older version. It should always be that the user can go back to the last configuration before the latest upgrade. But wasn't that the purpose of the fallback.img choice on the GRUB menu?

@Dan Yes, the standard kernel26 PKGBUILD is quite complex, but I'm not sure how avoidable that ends up being. If something could be done to work around the limitations of makepkg (and more appropriately, bash) your suggestion may well be a good one (since there would only be one PKGBUILD to maintain.) I wonder if there would be any other larger issues with doing it this way, or like you said before, if other packages (Python 2.4, 2.5, etc) might benefit from being able to do something similar. What I'm getting at is if only the kernel package is going to benefit from a feature/work around like your PKGBUILD, the first way I suggested with two PKGBUILDs may be a more appropriate solution.

@Mr. K. I don't think fallback.img will help at all if the kernel (for whatever reason) doesn't work. fallback.img is just an auto detected initrd that in most cases /should/ work on your system. So if you screwed up an option in /etc/mkinitcpio.conf you'll still be able to boot. For similar reasons, at least one backup kernel/modules is good to have around after a new kernel upgrade. Thus this feature request :)

On a side note, I'm in the process of trying my proposed method above to see how it actually works.

Ok, here's the example pkgbuilds for two kernels, a 2.6.29.2 kernel, and 2.6.30.1 kernel. It also includes kind of what I had in mind for kernel26 called mykernel26 (obviously it's not complete, just to give the idea, and both PKGBUILDs for the two versions are included.) The only thing that really changed between kernel-2.6.29.2 and kernel-2.6.30.1's PKGBUILDs was the md5sums, _basekernel, and pkgver.

The major things I changed from the current PKGBUILD was where the kernel version is declared, most of the time using pkgver instead of kernelname and removed most references to kernel26 in kernel26.install (not a full list, but there really wasn't much more then that.)

@Mr.K. Sorry for the delay. In noob terms what this should do is allow multiple kernel versions to be installed so if a kernel upgrade breaks something, they still have a known working kernel that can be used to boot the system (without the need for a LiveCD or install CD.) It is then left to the user to decide when they want to remove old kernels. (A feature to periodically remove old kernels may be useful, but it's not required.)

Oh, yes, very nice! You are so right. That makes a heck of a lot of sense.

In fact, I might add that I had always thought that was the purpose of the option during GRUB boot. If that isn't, then what is the point of GRUB? Who needs a menu when there is no choice. Since I found it wasn't the point, I'm lost with trying to figure out what it does. All very weird.

I've always been used to having a simple menu system that allowed me to boot into the partition I wanted to use. It didn't matter what partition, what OS or combination thereof the user wished to utilize. Then along comes the Arch install that reinvents a wheel that won't roll. I still don't understand the thinking behind this whole thing. How much better it would have been if we had simple gone forward from where we've been for fifteen years. One would add another item to their menu, and go from there. And if the package manager had installed a whole new OS kernel, then one should be able to assume that the fall-back system was standing by and ready to be used as a function of the package manager. We should have always been able to boot into a different (the old) version of an OS. All we need was for users to be able to step back to the previous, working setup. In this case, we don't even need a separate partition, since all is simple software replacement. Instead, someone went to a lot of trouble to make something broken by design. (How much more productive it is when we use our abilities to improve rather than reinvent. What wasted man-hours! But, hey, that's just my opinion.)

I've gone over the dialog here again. There is a huge amount of shorthand-speak that doesn't work in the real world, but I attempted to wade thru it. It appears that the issue can be simply stated as the need to make sure that a kernel does not have dependency problems. Is that the gist of it? Two points: 1. the kernel is supposed to be the lowest common denominator, so that shouldn't be an issue, 2. and in my case, all the upgrades I've done since that kernel was available have not caused any problem when I have reverted back to the older kernel. All is working fine now, including things that didn't work before. I have simply learned to avoid 2.6.30.6-1. I do realize that maybe that is just the chance of this combination of upgrades, but there may be no way to tell. It is at least worth trying another simple backup method.

Instead of changing pacman, why not make a simple reversion script? The purpose would be to take the last setup and put it back as it was. Not being sure of the exact inner workings of pacman, IMHO it may be that pacman would need to have an added function of logging the necessary details of the upgrade so that the added (or associated) reversion script would know what to do. This would be simple add-on script-like coding, rather than a complete rework of the program. I would definately not recommend anything that required any sort of reinvent.

In any case, thanks for the response and clarification. I'm glad I'm not alone in wondering about these things.

I'm a very new Arch Linux user (approximately one week so far :-)). After a kernel upgrade, I wanted to revert to the old kernel to see if the graphics performance degradation I was seeing was related to the upgrade. Coming from a long history of other distributions, I was somewhat astonished to find the old kernel had been overwritten instead of preserved. More than any other component in the system, a failed kernel upgrade has the potential to prevent booting. A simple reboot into an old kernel is easy and convenient in other distros, and Grub is tailored for exactly this kind of setup. If the various kernels could be treated as separate packages, it would be easy for a user to periodically use pacman to remove obsolete kernels. I don't know how much impact this has on the larger system, although many of the suggestions in the above comments sound reasonable to my Arch-novice ears. Perhaps Subversion tags could be used to mark the location of older kernel PKGBUILDS, allowing development to continue in the kernel26 directory. These "tags" wouldn't necessarily have to live outside of trunk; they could be dropped peer-level to the kernel26 directory, so they are easily accessible to users via abs. I don't really know enough to make too many implementation suggestions, though. But as a user, I'm very much in favor of supporting the simultaneous installation of multiple kernel versions.

I couldn't agree more. It is odd that there are so many unnecessary complexities within the subject that seem to create a cloud over the whole thing and stand in the way of progess. We really need to see this in simple terms. I can't help but recognize the truth in one of the things I've learned in the industry; which is that the end-user's impression is usually correct, and that, not only do these problems need to be addressed, but we need to use the KISS method in doing so.

I would suggest that those in the know first explain what is going on in creating the alternative boot setup that we currently have, so that all of us can understand where we stand now. I must admit that I still don't get it.

It sounds like we'd need to start storing kernels, initcpio images, and other relevant info in directories named for the pkgver and pkgrel. This starts to get complex, doesn't it?

I agree, it would be a great feature to be able to upgrade your kernel and trivially boot back into your old one if something went wrong. Can anyone lay out the full steps we'd need to do to make it happen?

How about this; let's leave it up to disk space. If we have plenty for the boot area, move the old kernel to backup directory. Change GRUB to add extra line. Install new kernel in normal location. If things don't work, one simply chooses last working kernel from GRUB menu. Then the user can run a script to remove new and replace old, changing GRUB in the process. Or something similar. Could symlinks be useful somehow in this process? (I'm just throwing these ideas out there for someone who knows kernels and their installation better than I and can look at these ideas and rearrange them until maybe something fits.)

Sometimes it might be useful to see the OS kernel as just another program to install. We know that it is possible to have more than one version of a particular program run in the same system. The complexity drops when we consider only one backup rather than multiple versions. It must be assumed that the old one was working and then new one was then added. That's only two. We've got that now, although useless, as another GRUB menu item. (Hopefully, we're not talking multiple boot partitions.)

Wouldn't a system like this cause dependency hell with making sure that the appropriate versions of third-party drivers are installed for each retained version of the kernel? All the various graphics and networking drivers come to mind.

Aren't the dependencies actually just a simple list of programs and their appropriate versions in a text file? Then wouldn't that simply be the source for the undo information by comparing the recent one with the similar file in the backup kernel directory? That should actually simplify the whole process so that a simple script can undo what was done.

The only thing we must keep in mind is that the kernel is separate from the other programs which may have been upgraded or installed at the same time. I believe that it is always important to upgrade the kernel separately from the rest of the system, so that the user can reboot and make sure it is working correctly before changing anything else. (I realize that this probably leaves a few things unsaid and assumed, but that's the general idea.)

Perhaps doing a "pacman -Syu" should note that a kernel upgrade exists first and do it separately with a required reboot notification. All other things should halt until this is done.

The alternative is that there is a lack of modularity in Linux, and such must be avoided at all costs. If it might be, as I suspect in this case, we begin to slide down that slippery slope, the whole system begins to be one humugous mess ala micro$oft. We cannot allow such a system to come into being. The issues of diagnosis alone dictate that this cannot be.

I think a full-blown versioned kernel is needed. If a kernel does not boot at all (maybe due to wrong config), then initramfs will not help. And if someone does not have a linux live cd/usb handy, then how will those people solve the problem? Simply overwriting old files with the files from the new package is okay even for xorg, kde etc. as the terminal/shell will still be available (init 3) to downgrade the packages? How will you get a working shell with a non-bootable kernel?

Since it is policy of Arch to make the user responsible for the configs, packages etc. it will not be a problem for an Arch user to manually update or remove kernel packages him/herself. Just make sure one kernel package does not overwrite another and leave the maintenance of /boot directory to the user. Arch devs need not worry about whether 1 kernel is installed or 5 or whether the /boot is small for many kernels . Let the user manage those things.

For versioned kernel installs, pacman should allow installing different version of same package. May be like ignore packages array, a new category of packages for which multiple versions can be allowed, should be created in pacman.conf and pacman source modified accordingly.

I like the idea of having the old kernel stay and not get deleted right away. The details of properly versioning files and such have already been solved by most other distros, so we could just borrow the ideas from them.

I completely agreed to Keshav P R, and i think it the kernel shouldnt be treated as any other program as stated before, because it is crucial for the system to work.

It was also stated before, that the user should not keep track of those things manually, but since arch-linux basically puts all decisions concerning updates to the user, i don't see why this shouldn't apply to the kernel. Maybe it would be possible to let pacman promt the user if he wants to keep old kernels once the user is cleaning the cache. (I mostly clean the cache when the disk is full and when my system is working like a charm)

Actually this problem keeps me from updating my kernel right now, because i need my computer to get some work done, and I don't wont to take the hassle to downgrade the kernel again if something goes wrong. I guess that applies to many users out there.

Sergej Pupykin
"I keep tinycore linux in my /boot for recovery" "Also I have kernel26 in pacman's IgnorePkg and never update kernel while I have no physical access to PC."
The fact that you do this and feel it as a required precaution should pretty strongly reinforce that this is an issue. It's even laughable to consider this (tinycore in /boot) a simpler solution than having a versioned kernel. IMO it's not a fix, it's a work-a-round to the real problem.

Now, if enough users/developers feel this idea isn't following Arch's philosophies close enough, I propose that the kernel26-lts be added to the base group to be installed by default. This still far from ideal, but it'd be a better than nothing. (Don't think of this as conceding defeat. I still strongly believe that versioned kernels are not only a good option, but can even be done in such a way that it follows Arch's philosophies.)

lonut
So the kernel26-lts wouldn't be suitable as a stable option for a kernel to recovery boot into upon issues? I hope it didn't look like I was implying for it to be the default kernel to be booted into, rather, a stable kernel that can be relied on to at least always boot. Though, if it's going to cause serious issues, it does nothing for the other case where a versioned kernel would be nice, kernel regressions and driver breakages.

Sergej
I never said anything about it being difficult. It the case of KISS, simple is not defined as easy, it's defined as lacking complexity. Again, I feel that your solution isn't a fix to the issue. It's a way to ignore the issue till it happens, and then work around the issue when it does. It also doesn't do anything for if there's a regression introduced. In that case, all the livecd's and rescue partitions in the world won't be of that much help (because the old kernel is gone.)

1. There is a kernel26 (dummy package) will always depend on the latest kernel version available.
2. Installing a kernel upgrade will rename all the old files (kernel26.img etc) to a versioned copy (kernel-2.638.2 os something) so that everythin is still the same to the old Arch user (except for the additional kernel image, which could be deleted by a simple script or tool if the user wants to).
3. Kernel modules will be placed in a versioned directory, which is not deleted anymore when a new kernel is installed.
4. Every time kernel26 is updated, a new package kernel-2.6.x.x is created that has the exact same contents of the old kernel installation. This is the most tricky part: removing this newly added (and automatically installed package, as it was installed as kernel26) should remove every file associated with it. For an example on how this scheme could work: I remember that using the 2010-05 Arch install cd also renames several packages when you first update with pacman. The same renaming can be used here.

Items like /lib/modules/<version>/ would be pinned by the version-specific package, forced by the depends.
You only need to update the vmlinuz26 and kernel26.img hardlinks as they are pointed by grub (you could as
well try to update its reference, but as that may be edited by the user, given how cheap hardlinks are,
it seems easier to do this).

Comment by Anonymous Submitter -
Monday, 06 June 2011, 15:45 GMT

Now that officially the next kernel will be 3.0 , I think it is a good time to remove kernel26 naming altogether (aka reconsider implementing any one of the proposals) and move to version specific naming. This will also require change in naming of the initramfs image corresponding to the kernel https://bugs.archlinux.org/task/18719 .

I think it would be nice if pacman could allow you to install multiple versions of the same package. Now for most packages you wouldn't be able to install multiple versions, but for packages like the linux kernel, and python this would make sense. This should allow making it so that version x of a package doesn't conflict with version y but version z does conflict with version y. For example you have python/2 which doesn't conflict with python/3 but python/2.6 does conflict with python/2.7. I think you could have the syntax in the PKGBUILD like follows:

conflicts=(!python=3.*)

The ! means, "doesn't conflict". The conflict for the kernel26 could look as follows:

conflicts=(!kernel26 kernel26=<3.0)

The above would mean, we don't conflict with any kernel26 version which is more than 3.0, since the older versions of kernel26 might conflict as they were not designed to work with the newer multiple versions system.

This would not only solve the issue for multiple kernel versions, but it will also simplify multiple other packages such as python. The future "linux-lts" package could simply just require a "linux/lts-version" package. The package repositories will need to support multiple versions of a package also, so python/2 and python/3 could be in the repo.

The only issue with multiple kernel versions is with boot loaders because the kernels would have to be in different locations to not conflict. The kernels would be placed somewhere like "/boot/kernels/linux-*" for each version. From what I understand is that there could be a grub2 script or even a grub2 module which could create the boot menu from different installed kernels inside the "/boot/kernels" folder. This would only support grub2. Grub1 and LILO users would need to manually change there configs to reflect new kernels.

For support with LILO/grub1 there could be a package which trials the latest kernel and provides a symlinks and depends upon the real kernel. A user would have 3 options, use grub2 and the automatic menu's, use the kernel trailing package or edit the config themselves. The trailing package could be something like "linux-symlinks", and the new linux package called "linux" (once 3.0 is released).

The only issue is that dependencies would have to support a new syntax, so you can provide a dependency to a certain set of versions such as python2-bla package would depend on python=2.*, if you have python/3 installed the installer should detect and then install the python/2 package alongside python/3.

The package repositores can continue to store python2/PKGBUILD and python/PKGBUILD but the package name inside the package would be the same "python". The linux kernel could adopt a similar pattern.

If a few people like this then I will file it as a complete new bug/feature report for pacman. I have tried to explain this as clearly as I can, I hope it makes sense.

@Kyle: Just one point left, how is the user supposed to install the different packages as they are now named the same? I could imagine (but I doubt this will be real) to implement a shallow kind of regex into pacman, so it can easily match pkgname:ver as input (for any kind of versionen sheme). But your effort throws up many questions. Should multiinstall of packages then be enabled for all pkgs? Should we introduce a pacman flag to allow duplicate install? How to handle multiinstall for files in /usr/bin/ /usr/sbin/ /sbin/ and so on...

Dan and I had a short informal talk recently, and it looks like we can solve this cleanly without having multiple packages of the same name. This requires some changes to the linux package, these changes will probably be done slowly starting with the 3.0 release - most importantly, the different kernel versions must not have file conflicts.

It also requires support in pacman, so that we can automatically remove old versions. This will all go slowly and take a while, just wanted to let you know that we will start a process that will eventually lead to the resolution of this bug report.

This is not an acceptable solution. This will lead to files being in the system that are not tracked by pacman and have not been installed from any package. This is an absolute no-go and we will never do this (yes, NEVER).

The kernel file should also be versioned (e.g. /boot/vmlinuz26-(ver), and make /boot/vmlinuz26 a symlink to the latest). Thus, two kernel packages would not conflict each other at all except over that one symlink file. Pacman can then have a mode that keeps two packages around so long as the only conflicted files are symlinks (in which case, the symlink is taken from the more recently installed or higher versioned package). The user then manually edits the grub.conf to add older kernels as options. This I think is the KISS approach.

If we want to get fancy, then in addition to the above, we can do something similar to RHEL which updates the grub.conf at every kernel install. Grub would need to have a menuentry auto-added for each installed kernel (and the menuentry auto-removed when a kernel is removed), also like RHEL. The default menuentry would continue pointing to vmlinuz26 (the symlink), only additional grub menuentries would point to specific kernel files. This approach requires zero manual intervention from a user, it will "just work" -- when they reboot, they'll see menus for the latest kernel as well as entries for all the specific versions they kept around. I prefer the KISS approach of not auto-managing grub.conf, but at least creating the kernel packages such that they never conflict with another kernel package except over symlinks.

Marti, those are special files and should probably also be handled by pacman somehow, but this shouldn't be the topic here.

Okay, before tons of bikeshedding starts, here it is: Eventually, each kernel package will have a kernel version like 3.1.X-Y-ARCH where X is the upstream patchlevel, Y is the pkgrel and -ARCH indicates that this is the Arch Linux kernel. This will take care of /lib/modules/* not conflicting. Kernel image and initramfs will have this version appended to its filename.

Now, the packages: The 'linux' package will depend on the following packages: linux-version-3.1.X-Y (this is the pkgname, without version) and linux-version-3.1.X2-Y2. The 'linux' package will also manage symlinks like vmlinuz-current and vmlinuz-old. Now, we need a new feature in pacman: we need to be able to obsolete a package so pacman uninstalls it on its own.

Anyway, this approach is not entirely thought out yet, I still see a problem when someone skips a few kernel versions, and with external modules. As I said, we will transition to this slowly rather than in one step.

@Kyle: Just one point left, how is the user supposed to install the different packages as they are now named the same? I could imagine (but I doubt this will be real) to implement a shallow kind of regex into pacman, so it can easily match pkgname:ver as input (for any kind of versionen sheme). But your effort throws up many questions. Should multiinstall of packages then be enabled for all pkgs? Should we introduce a pacman flag to allow duplicate install? How to handle multiinstall for files in /usr/bin/ /usr/sbin/ /sbin/ and so on...

Thomas, Just thought i'd throw an idea out there. A script that waits until the nth successful reboot to remove the old kernel. Execute the script through the rc.local. Simply, a successful boot counter. When it's save to proceed, remove the old kernel and the script from the rc.local, until of course next kernel update. That also solves your pacman issue, as the rc.local is launched from root at boot. "uninstalling it on its own".

@dangbarry: I don't think that's good idea. System might not work as expected completely, and the script would increase counter. While user tries to debug it, script will uninstall the good older kernel.

I recommend that there is a config option in pacman.conf: KERNEL_VERSIONS or something simmilar. This should specify how much versions of kernel should be keept.

For example, let's say it's 3. When first 3 kernel versions install, nothing happens. But when 4th kernel update is installing, pacman should remove the first one. This could be done using a workarouond like this in .install file:

# do a standard depmod, mkinitcpio, create symlinks for new kernel to be run as default
#

# move the pacman lock so another instance of pacman can be run.
# This is save because one pacman is just waiting for the second one to finish, so pacman database is safe.
mv /var/lock/pacman.lock{,.bak}

A user who manually installed these packages will not expect them to be removed automatically, but they would be. The other issue is that this may break if linux-lts uses the same system for versioned kernel installs (which it should).

Imho, this is pretty easy to solve. Either we do it the Debian way with lazy cleanup or we keep all kernels and have the dummy package depend on the real kernel as described in the first three comments and keep the old packages.

Imho there is no harm in keeping them, we should probably consider a feature like "autoremove" in apt, which removes previously needed, but now unneeded packages, with a hint anyway.

There maybe the problem that a user runs autoremove after installing a new kernel and then removes the current kernel, but I guess this shouldn't be a big issue for arch users.

I know there's a lot of opinion here and that isn't always productive, but I think the solution is this.

1: Switch to GRUB2. GRUB 0.97 is undeveloped, patched like nothing else and is dead in the water. GRUB2 has grub-mkconfig and will detect everything in /boot, even Windows partitions on another disk with os-detect.
2: Keep one linux package that generates a different initial ramdisk upon installation/upgrade based on whatever version number you want. This will keep the old initial ramdisks available for use since they're not installed with the kernel, they're generated by mkinitcpio.
3: Call "grub-mkconfig -o /boot/grub/grub.cfg" in the kernel upgrade to re-generate the file. If it overwrites the file and someone has manually customised it, shame on them, because thats not how you modify GRUB2. If a user doesn't have GRUB2, the call will simply fail and the user can update their bootloader. If they're not using GRUB2, then they're probably doing something else, so they'd know what they're doing.

I think that's pretty KISS. I personally don't care about 3, because I know Arch is a bit more manual than others and I have no problem with typing a one-liner.

As a sidenote to Sergej, you say "Also I have kernel26 in pacman's IgnorePkg and never update kernel while I have no physical access to PC". Next time you're buying a PC/Server, get something with iLO, you'll have remote access to the entire system with actual video output, including full system BIOS access :).

Yes, it seems everyone needs to give an opinion here, and none of it is productive.

@Mitchell: As for your opinion - I don't quite understand what you actually want to do, but you seem to think having a different bootloader will solve some problem. You also seem to think that now everyone uses grub and later everyone will use grub2. You also seem to think that regenerating a configuration file on each update using some voodoo script that we have no control over is a KISS solution. Whatever it is you're suggesting, it won't happen.

I am thinking about simply closing this report. It's going nowhere, everyone just posts new "solutions" that we simply won't implement. I posted a game-plan in an earlier comment, which will eventually be implemented. I consider this very low priority, and thus the required changes will be made slowly. Leaving more and more comments with brilliant "ideas" here (which spam my inbox with bugtracker emails) will not speed it up.

By all means, my opinion is just that. By having grub 2 as a default allows for easier bootloader updates, yes. I stand by having it as a default because it's what it's upstream developed and supported. This it's of topic though.

What I'm ultimately saying is to have versioned initial ramdisks, then new initial ramdisks can be generated upon kernel upgrades.

Lastly, no voodoo script by any means. I am only talking about one line to generate a new config for grub. It wouldn't be any more code than what's used to generate the initial ramdisk upon a kernel upgrade now.

I don't assume everyone to have grub as a bootloader at present, no. but those who don't follow defaults and want to do their own thing would be smart enough to deal with updating their bootloader.

Comment by Thomas Bächler (brain0) - Thursday, 14 July 2011, 14:50 GMT
Dan and I had a short informal talk recently, and it looks like we can solve this cleanly without having multiple packages of the same name. This requires some changes to the linux package, these changes will probably be done slowly starting with the 3.0 release - most importantly, the different kernel versions must not have file conflicts.

It also requires support in pacman, so that we can automatically remove old versions. This will all go slowly and take a while, just wanted to let you know that we will start a process that will eventually lead to the resolution of this bug report.

Could i ask if this particular work is advanced? Just for have a general idea on when this function will be available?

This is an issue, but I don't think it's really a big deal.. it really depends the way you look at it.

What I do is I just copy the current kernel with a different name. For example we are currently (at the time of this post) on 3.4.2-2-ARCH.

So I do a `cp /boot/vmlinuz-linuz /boot/kernel-3.4.2-2-ARCH` and `cp /boot/initramfs-linux.img /boot/initramfs-3.4.2-2-ARCH.img` and add the entries into my bootloader. Thus if I ever want to stay on that kernel, I can reboot into the old kernel (or set it as default).

The above method doesn't solve the "Temporarily unable to load my drivers in between kernel upgrades" but it does solve many other issues.

I would love to see this feature implemented.
Recently, a kernel upgrade from 3.7.10 to 3.8.3 broke my co-worker's Arch. We had to use a usb drive, unlock LUKS, detect LVM, mount drives, chroot, reinstall 3.7.10.
All would've been simpler, had we the ubuntu-way of choosing older kernels in grub.
Please, think about it.
Thanks.

In the meantime, I'm quite fond of keeping an ArchBoot kernel & initramfs around for quick access to a rescue system. You can add an entry to your bootloader, and so you always have something to fall back on :)

We still need a way to install each kernel version under a different file name and automatically update the boot loader afterwards, but I don't see a solution for this (the freedesktop.org bootloader spec is a start, but 1) you'd lose custom kernel arguments on each update, 2) only gummiboot and a patched grub even support the spec right now).

I am booting from UEFI. /boot is ext4, and /boot/efi is FAT. The kernel is in /boot, so a symlink would be fine for me. I have /boot/grub/ symlinked to /boot/efi/efi/grub/. Slightly convoluted, but it works great.

Only works with grub. Grub is the only bootloader that can load kernels from other partitions than the ESP. With all other bootloaders, your kernel must be on the ESP where the firmware can access it directly.

This has already been suggested. The problem is that some people have a FAT filesystem for /boot, which doesn't support symlinks. It might be possible to get around that by simply copying the file, so long as the filesystem isn't too small.

In my opinion, the linux-lts package covers this use case well. The ESP partition is usually *really* small and we are going to cause a lot of problems if we fill it with old kernel versions. This is something that users can do if they really want, but I find it to be an overcomplicated, overly magical hack for the package to be involved with an install script.

Additionally, @phrakture and I cannot be the only users irritated by such things as USB storage breaking upon upgrading the kernel. This is due to the old module directory being purged immediately and thus requires a reboot before new devices can be hotplugged.

I propose 3 possible solutions to this annoyance:
a.) Versioned/slotted kernels
b.) Build core drivers into the kernel (why for instance are ext4, btrfs, etc. a module?!?)
c.) Create a hook to optionally keep the old modules directory UNTIL reboot.

I think we need to be specific what is really needed:
* modules for newest installed kernel
* modules for other installed kernels (if allowed)
* modules for current running kernel whether installed or not

It's not only about USB, it's also about all on-demand modules, including firewalls, virtualbox, etc.
Typical scenario is:
* boot, work a bit
* pacman -Syu
* start virtualbox, fail

On the topic of LTS kernels supported on kernel.org, I recently got a repository going for them at tiny.cc/clfarron4
But that's not the issue here and it doesn't help solve the one proposed here when you consider that the current version of the kernel (3.17/16) and at least two of the LTS releases (3.14 and 3.10) get updated at the same time.

An alternative solution is to implement systemd offline updates[1]. I would be nice if it only updated packages that need to be done offline, offline. The only one I know of is the kernal, but there may be others. This will require some small additions to pacman.

Well we still haven't solved the problem of the running kernel's modules being removed when installing a new kernel... This can cause issues e.g. when plugging a USB keyboard into a server after remotely installing a new kernel. The kernel is unable to load the relevant module, and the keyboard is unusable. It's quite a confusing situation for someone who doesn't realise why it's happening.

LTS for fallback is also not sufficient if both the normal version and the LTS-kernel was updated at the same time (or at least between your updates). You could of course manually postpone upgrading the LTS kernel, but that makes upgrading more convoluted.

Another issue which might affect some users is that hibernate (to disk) breaks if you boot a different kernel than then one which performed the hibernate. i.e. you upgrade kernel, hibernate, and then when you power on the new version of kernel doesn't manage the loading well and crashes. This was at least true somewhere at the beginning of 3.x, I haven't tried in a long time.

Coming from Debian, I am used to having multiple kernels installed. Having a known working kernel is important in case something doesn't work with the new kernel.

As I have my Arch install on ZFS from the AUR, I wouldn't want to be without my data due to some possible issue with modules not building, or being unavailable until a reboot. (I may go months without a reboot.)

I have currently installed a kernel from the ABS. I've simply edited the PKGBUILD to change pkgbase=linux to pkgbase=linux-4.1.6-1, and built the package(s) as usual. The result was being able to install a new kernel and headers without overwriting the old ones. I've also created an empty dummy linux package for compatibility/dependencies.

Now obviously that means that building new modules such as ZFS means pointing the dependency to linux-x.x.x instead of just linux. I've also changed the module names in PKGBUILD to match the kernel it was build for (zfs-git-4.1.6-1). This means I can, if necessary, boot into the older kernel, and still having working modules for it too.

I know this sounds like extra work, but we use Arch because we aren't afraid of that right??

Perhaps instead of having AUR packages look for a specific version of the kernel, why not specify the kernel to build against during makepkg? (I had to fake uname -r to report the new kernel instead of the running kernel.)

systemd comes with the `kernel-install` tool, which takes a kernel version and image as argument and then runs /etc/kernel/install.d and /usr/lib/kernel/install.d on it. The latter contains 50-mkinitcpio.install, which compiles a initramfs for the given version and places it in /boot/$machine-id/$version/initrd, as well as 90-loaderentry.install, which will copy the given kernel image to /boot/$machine-id/$version/linux and create a versioned entry in /boot/loader/entries. This is, as far as I understand, what you want.

from an end user's perspective: I'd be in favor of anything (within reason) that doesn't require manual removal of old kernels. Some other well known distros will happily fill up the boot partition until a novice user can't update anymore and dependencies are borked. I understand arch is not for novices, but if a piece of software can install multiple kernels, it can clean up after itself too. Requiring pre config is fine. Periodic manual intervention, not so much. The current way arch handles it is something i've been happy about for years, though i have seen issues with modules so i understand trying to fix that.

I suggest to add "old" suffix to the old kernel and initramfs files for the last kernel. So that, when a new kernel update appears, there will be vmlinuz-linux-old, initramfs-linux-old.img, and initramfs-linux-fallback-old.img that can be booted after updating grub.cfg entry. This can be triggered by libalpm hooks (pre-transaction and post-transaction).

1. Each kernel version has its own version-named package as described above (e.g. linux-1.2.3.4-1.2.3.4-1 where linux-1.2.3.4 is the pkgname and the second 1.2.3.4 is the pkgver),
2. Each one of these version-named packages will "provide" the linux-real=${version} package. That is, although each version-named package looks like a completely different package, it also provides a version of a common named target (i.e. linux-real),
3. A dummy package like linux-virtual-${version} will be created which always depends on the latest linux-real-${version} package and also contains the above mentioned symlinks. This installs the new kernel version as its dependency and leaves older versions installed as unused dependencies,
4. The linux-virtual dummy package also "conflicts" with the "linux-real<=${N older version as you wish}" and so all older kernel versions which provide those older linux-real targets are found by pacman and will be removed after getting a y/N confirmation from the user.

This requires no special maintenance script for removing old kernels nor any special support from the pacman.

I'm surprised nobody has mentioned this yet, but since Linux 4.0, kGraft and kpatch were included in the kernel. Not to mention ksplice, which has been available for years. While having multiple kernel versions installed seems important right now, I think that live kernel updates is the future.

Live updates are aimed at making small security fixes. It's not a way to do arbitrary upgrades, and doesn't simply Just Work without work to create the live patches. I also don't really see how it's relevant to keeping multiple kernel versions installed...

We now have 3 official kernels in our repos: stick kernel, -lts and -zen. I'm for closing this bug
as nobody seems to be interested in implementing this and there's no real need to have fallback kernels of
the same branch.

While there might be no real need for YOU to have a copy of the running kernel in the filesystems, there is definitely a strong need for others.

E.g. read my comment above (https://bugs.archlinux.org/task/16702#comment143419). When you have a kernel in /boot and EFI is setup to start it (e.g. using gummiboot or systemd-boot), but the /lib/modules/${KV} directory associated with it is gone, the boot will fail. There will be no option left to boot into the system and one has to boot from another medium (e.g. usb drive) to fix this up by copying the newly installed kernel image into the right place (the place expected by EFI/gummiboot/systemd-boot).

i think this bug report is not really about having lots of different kernels installed simultaneously. it's about retaining the running kernel and its modules. otherwise this can happen:

1) boot up kernel X
2) update to kernel X+1
3) plug in usb stick/mount some filesystem/try to load any module not yet loaded
4) cry after you find out old modules (for the running kernel) are gone and you need to reboot

I can implement it. It is not a very complicated task.
Do we have any specification on how it is supposed to be implemented?
If not, then what i can do is:
- a systemd init script which copies the current modules into /lib/modules/current + cleanup of the old symlinks to current
- a post uninstall hook for the kernel which will detect and symlink if necessary 4.6.4-1-ARCH/ to current/

Okay, before tons of bikeshedding starts, here it is: Eventually, each kernel package will have a kernel version like 3.1.X-Y-ARCH where X is the upstream patchlevel, Y is the pkgrel and -ARCH indicates that this is the Arch Linux kernel. This will take care of /lib/modules/* not conflicting. Kernel image and initramfs will have this version appended to its filename.

Now, the packages: The 'linux' package will depend on the following packages: linux-version-3.1.X-Y (this is the pkgname, without version) and linux-version-3.1.X2-Y2. The 'linux' package will also manage symlinks like vmlinuz-current and vmlinuz-old. Now, we need a new feature in pacman: we need to be able to obsolete a package so pacman uninstalls it on its own.

Anyway, this approach is not entirely thought out yet, I still see a problem when someone skips a few kernel versions, and with external modules. As I said, we will transition to this slowly rather than in one step.

As far as I can see, this can be done today, as long as users are responsible for uninstalling their own kernels (which is not an unreasonable idea).

Do we just need (questionable) support in pacman for obsoleting a package according to user-defined rules?
If so, has anyone opened a ticket against pacman and/or discussed it on the pacman-dev mailing list?

Because an excellent way to make sure it never happens is to forget all about it and then say no one is interested in it because no one is sure what they actually want.

Yeah, for your arbitrarily chosen target users and your arbitrarily chosen number of manually-specified older kernels, which kind of defeats the purpose.

... I still don't see what is wrong with requiring users to manage their own orphaned packages, in which case this can be implemented as soon as someone decides they want to do it.
But if automatic purging of old kernels is what is stopping this from being implemented, it would be nice to at least know that that is the reason.

It seems to me that people are trying to address two different (though similar) problems:

One is the inability to have multiple versions of the kernel at the same time. This is a problem in case you're not sure the new version works, and want an old one to fall back on.
The other is the inability to leave a copy in the filesystem of only the *currently running* kernel and modules. This is a problem for loading new drivers after an upgrade, or for hibernating and resuming system with the same kernel.

In order to solve the first problem, something akin to having a different package for each kernel version is required; and then there is the question of how to manage the old versions, via package conflicts, obsolete package feature for pacman, or just leaving it to the users.
In order to solve the second problem, a script which saves the current kernel and modules at init, perhaps with a bit more (see babali's comment https://bugs.archlinux.org/task/16702#comment149179) is enough.

Now, the question is, which of the two problems are we trying to solve? It is true that a solution for the first one also solves the second one, but is probably more complicated (hence the stalling).

i totally agree (with solving just the second problem). ...adding the need for manual uninstalling of old kernels would be a step back, imho. (for example, many of my systems are quite limited on disk space)

I'd like to
1) have all new kernels installed by default (i.e. have the linux package depend on the latest real kernel package).
2) have a post-install hook which will remove kernel packages, with a configuration file to specify how many releases of the kernel it should keep. It should always leave both the latest and the current running kernel in place.
What do you think?

Automatic uninstalling of old versioned kernels requires either ugly usage of the conflicts field, or else extending pacman. (Post-install hooks cannot invoke pacman, since the database will be locked by the parent pacman process.)

Copying the kernel to outside of pacman's bookkeeping is ugly and not simple at all, besides which in order to actually use it in any way you'd need to copy it right back. In other words, you are literally trying to trick pacman into thinking a package is uninstalled while leaving the files exactly where they were.

...

As per the original intent of this ticket, the solution is *still* versioned kernel installs.
So-called "simpler" solutions are in fact more complicated, and IMHO against the Arch philosophy.

So, it would be nice to hear from the kernel maintainers if "automatic obsoletion of old kernels, possibly according to user-defined rules" is the only blocker.

I filled a feature request for the pacman package (https://bugs.archlinux.org/task/50121) which allows installing the latest kernel version while keeping the current kernel version even if the user has skipped N (unknown) kernel versions.

Not all discussion is bikeshedding. Of course a plethora of solutions, minutely different from each other, can be suggested and would get us nowhere; but I think that discussing the options we have broadly, not to mention the requirements of a solution, does help dispelling our current "no one is sure what they actually want" situation, to quote you. If we manage to break down and formulate a few (two-to-three) options with their merits and downsides, then it will be much easier to choose one of them and implement it. It seems to me that we have now two main (categories of) approaches, and what we need is to flesh them out.

I agree that copying files outside of pacman's bookkeeping is not the most elegant of solutions, but it is a possible one. There are precedents for unowned symlinks, such as /etc/localtime, admittedly more in configuration than in binaries. Having a temporary copy of the current modules doesn't sound that bad, perhaps it can even sit outside /usr/lib/modules.

That said, I agree that having multiple kernel versions is definitely also a reasonable solution, and that Arch users should be able to manually manage them. At the very least, it is also possible to have a linux-latest package, so users which don't want to have this feature can go on with the current way of things. So, I second the question whether outside of the obsoletion managament, there are no problems with implementing this.

You could install multiple kernel packages, each carrying the version in their name, and then use something like Gentoo's eclean-kernel, which is a script that scans /boot for old kernels and removes them. With Arch Linux binary packages of kernels you could instead just call pacman -R from that script.

This should be rectified by causing the modules in the linux package not to be removed on upgrade if they correspond to the running version. Instead, the module directory should be marked for deletion. A script running at boot can then remove any old modules marked for deletion where they do not correspond to the running version.

Primary need here is access to a recovery console in a catastrophic case right? Coming to mind after #51818 today...

Fix here seems simple using pacman builtin backup. pacman manages backup history depth already and $(pacdiff -o) will flag the offending files for the clean system purists in us all. No less elegant than the way /etc/pacman.d/mirrorlist is handled -- well, excepting that these are binaries. These would be natural pairs having been owned by the same package before orphaning.

Bootloaders then need to know how to find these and register them. Looks like grub already does in 10_linux through flagrant wildcard use. Otherwise, light sed/awk lines in the PKGBUILD can clone out for finding .pacsave suffixes and adding below the primary boot cases.

EDIT: The first word in this report is "Versioned" and I acknowledge I ignored that, but it's not an altogether inelegant idea despite being admittedly limited.

No, backup is not the right approach. Firstly, it only applies to modified files, so you'd never get a backup of the kernel. Secondly, the initcpios aren't even part of the package (and can't be sensibly made so), so you'd never get a backup of the them, either.

As this issue has been closed with reference to this task (which is ok) and a thoroughly shitty comment (which is not ok), I move here.

If I understand correctly, this issue is in "undecided" state since 2009. Looking at the technicalities, I am not sure why.
The Linux Kernel deployment mechanism is such that several kernel versions may be installed in parallel on a given system.
Why does Arch not make use of that?
Where is the insurmountable problem of having a (configurable) keep last N kernels option, where N can be - say - 3?
What problem is there to at least provide a compatibility symlink for same kernel versions of an older patchlevel?

And finally in reference to the aforementioned shitty comment in task 51873: So if Arch decides to update the Kernel package it really is considered a "bug" on the users side to not immediately reboot? Srsly?

Pacman does not support that use case. You cannot install multiple versions of the same package. Putting a version into the package name causes a large maintenance burden on the side of the repository (the repository management being nearly unmaintained is the reason we're still stuck with SVN).

There is no guaranteed compatibility even between kernels of the same minor version, so allowing new modules to be loaded into an old kernel is unsafe and may cause crashes or irrecoverable corruption. This has bitten us in the past.

Yes, after a kernel upgrade you need to reboot if you want to load any modules.

Quote: "An additional number (now the third number) would be added on when necessary to designate security and bug fixes, as for example with 3.0.18"

You will never have API, IOCTL or otherwise binary incompatible changes with different minor versions. Show some counter-example.

I moved my notebook from Gentoo to Arch to test how things go here. Not having to compile is nice. But do I understand correctly that there are no different versions (ok) and not even "Slots"? of packages?

IIRC the whole Nvidia-OpenCL stuff pulls in a gcc5, while the system has a gcc6?

So no way to have PHP 5.x and PHP 7.x at the same time?

And no, I cannot accept to have to reboot after a 4.8.8-1 to 4.8.8-2 "upgrade" (you may want to look up the meaning of "upgrade")

KConfig changes can break ABI. Any package release might change the configuration, even if the version remains the same.

There are no slots. There can be at most one package with a given name installed. You can have multiple versions of the same software, but you need to give them separate names. As far as the repository and package management is concerned, that makes it separate software.

"it's more work to maintain" is a poor reason to waffle on supporting this use case. If better repo management tools are needed, I am more than happy to lend some spare hours working on improv(ed|ing) tools and/or package install scripts if implementation of improvements is the only thing holding back a resolution of this usability wart. Having a known working backup kernel is not the only use case where having multiple kernels installed can be desirable.

In the Debian/Ubuntu world, `linux-image...` is a metapackage that always points to the current kernel. This allows for tracking new kernels without immediately removing the old one, and users/admins the freedom to reboot when they choose to, as there are no slots in that packaging system either (and authoring a deb is a lot more work than writing an equivalent PKGBUILD). This could be an acceptable mechanism to implement this without needing to make significant changes to pacman and packaging tools.

pacman is capable of not removing files it installed on install when a package is removed, like any package manager. Simply mark kernel modules as such files and use a hook script executed on boot/shutdown/after pacman execution which cleans up any modules which match neither the installed package version nor the currently running kernel version.

The problem with your solution is that it doesn't actually work. pacman is not, in fact, capable of *not* removing files in a package when it uninstalls the package. Not unless you extend pacman first, by which logic you can do a lot of things. And as has already been noted, extending pacman itself for the specific use case of the linux kernel is ugly.

@heftig,

"Putting a version into the package name causes a large maintenance burden on the side of the repository (the repository management being nearly unmaintained is the reason we're still stuck with SVN)."

What exactly is this maintenance burden? I cannot say I have personal experience, but I would assume the repo management already knows how to handle split packages when the pkgname array changes from one version to the next. Unless you are saying that when part of a split package is removed from the PKGBUILD, it falls out of sight and manual work is required to purge it from the repos? In that case, maybe this is an opportunity to fix something. :p

Saying "this particular solution won't work because the repo management is unmaintained" is, as @Commod0re said, a poor reason to waffle, but moreover, it is essentially saying "we have no intention of dealing with this bugreport in any way, because it involves WORK".
So it is unmaintained -- if you (rhet.) are motivated to solve this bug, you can one-off patch the repo scripts, and then let it languish unmaintained again. Unmaintained just means nobody cares about it, not that it is impossible to work on it.

Is that the blocker then? (Were you wearing your Arch Dev hat when you said that?)

...

Or, does anyone know what the blocker is?

I really want to hear from someone who can say "this is what we need in order to have versioned kernels, if you implement it I will push in support of finally fixing this".

Going over the comments again, it seems there are two serious concerns raised by the people actually in charge around here.
@brain0 mentions earlier ( https://bugs.archlinux.org/task/16702#comment80122 ), that he wants a way for pacman to obsolete and remove old versions automatically. Additionally, the bootloader would need to be reconfigured on every update because not all bootloaders support symlinking a *-latest kernel/initramfs. (I am looking at you, UEFI/ESP on stupid FAT!)

Old versions IMHO are the user's responsibility, but an argument could be made that stuffing the ESP with obsolete kernels is a bad idea. ;)
Grub could ship a pacman hook to update via grub-mkconfig after (versioned) kernel installs -- this might not be a terrible idea anyway, and users are as always free to disable hooks they don't like. So much better than in the install script which can't be disabled!
I don't know about other bootloaders, how easy it is to update automagically.

pacman seems perfectly capable of not removing config files. Or just ship with a directory /lib/modules/$foo.tmp and use a postinstall script to rename it to /lib/modules/$foo, or similar, preventing pacman's removal process from finding it.

1) You are not comparing apples and oranges, you are comparing apples and decorative china knicknacks. pacman knows how to leave alone any files explicitly marked as backup files, in the event that they have been modified from the original packaged version -- which is how it handles user-editable config files. Non-edited files are deleted as usual, and in any event marking 4822 files as backup files would be a terrible idea even if it worked, which it doesn't.

2) If you really want to install a kernel outside of pacman, there are plenty of ways already. Any solution MUST involve pacman continuing to know that the files exist, or else there is no point of using package managers and we might as well go back to the old days of installing software via `./configure && make && sudo make install`. Or untar'ing the *.pkg.tar.xz into / so it is present on disk but we don't have to worry about pacman knowing about it.
Untar'ing it by hand actually works really well, if you just want to cut it adrift from pacman, why keep around a dummy package which doesn't do anything?

To add insult to injury, pacman would get all worried when about 5000 files drop off the face of pacman, and end up being profoundly irritating.

...

Both those ideas were rejected already, and not only because they have nothing to do with the actual request of this task.
If you have any more horrible and ill-thought-out ideas, please don't mention them. Your comment is the poster child for @brain0's previous comment:

"I am thinking about simply closing this report. It's going nowhere, everyone just posts new "solutions" that we simply won't implement. I posted a game-plan in an earlier comment, which will eventually be implemented. I consider this very low priority, and thus the required changes will be made slowly. Leaving more and more comments with brilliant "ideas" here (which spam my inbox with bugtracker emails) will not speed it up."

Might I add that the aforementioned brilliant "ideas" have already long since driven that particular Arch dev to unsubscribe from this task? Good going, everyone, hope you are happy now you had a chance to talk...

I, personally, just want to know what the status is of the official, blessed "best solution", and how that can be brought to fruition faster if possible.

LOL ... just LOL.
One month in Arch and already great cinema.
You know - if you "all" want to have a "task" open for over 7 years and dancing around why a solution is not possible, it's actually fine with me.

I'm perfectly fine to keep Arch contained on my notebook only and spare the other dozens of machines that are on a rolling release distro. I also can endure this little hassle, because for me, it is either putting the linux kernel in IgnorePkg=linux or

Because after it upgraded to 4.8.10 yesterday and NFS stopped working and you really F***ING expect the user to reboot, then you have a distro that is worse than Windows95. But hey - as long as *I* can fix it, no hard feelings.

> 2) If you really want to install a kernel outside of pacman, there are plenty of ways already. Any solution MUST involve pacman continuing to know that the files exist, or else there is no point of using package managers and we might as well go back to the old days of installing software via `./configure && make && sudo make install`. Or untar'ing the *.pkg.tar.xz into / so it is present on disk but we don't have to worry about pacman knowing about it.
>Untar'ing it by hand actually works really well, if you just want to cut it adrift from pacman, why keep around a dummy package which doesn't do anything?
Old modules can get cleaned up by the cleanup script, not the package manager. There's no particular reason why we should be bothered as to whether it's pacman or another mechanism managing the lifetime of certain files, and many files on the system with managed lifecycles are managed by other processes. The proposed solution provides mechanisms which ensure that all of the files exempted from package manager cleanup are cleaned up by other means. There is no leakage here. No currently available solution, or your suggestion of just installing a kernel manually, offers this.

>To add insult to injury, pacman would get all worried when about 5000 files drop off the face of pacman, and end up being profoundly irritating.
What on earth does 'worry' mean in this context? This is not scientific language.