The mustard indicates progress.

Main menu

Post navigation

fedup: a little background

A short history of upgrade tools

In the beginning, the Installer was all. If you wanted to upgrade a Red Hat / Fedora system you downloaded the CD images, burned them all to CDs, and sat around swapping disks in and out until your system was upgraded. What fun!

Five years ago I was a QA guy, doing lots and lots of upgrade testing like this. Eventually, after burning a couple thousand CDs, I had this idea: hey, it’d be cool if you could upgrade your system by running some command that would set everything up for you and do the upgrade, without having to download all these ISOs and burn them to CDs. A small pile of gross hacks later, it turned out it was actually possible… and thus PreUpgrade was born.

And so it was decided. So the new installer doesn’t handle upgrades. At all. Something new is required. And that something is..

Fedora Upgrader – “fedup” for short

The premise is simple: There’s a frontend bit that sets up your system for the upgrade (downloads packages, sets up kernel & initrd, modifies bootloader entries, etc.). That’s fedup. And then there’s the “upgrader image” – the thing that fedup downloads to actually run the upgrade. That’s fedup-dracut. As the name implies, it’s just a regular dracut-built initramfs, with some extra bits and pieces thrown in.

The process works something like this:

fedup

downloads packages and kernel + upgrade.img from the new release

sets up some directories and files so fedup-dracut knows what to do

sets up the bootloader config so it will boot the new kernel + upgrade.img

reboots

fedup-dracut

starts the system and finds your root partition (like normal boot)

enters your old system and sets up the disks (like normal boot)

systemd trickery: If the files that fedup set up are present, we return to fedup-dracut and run special dracut hooks:

upgrade-post: for cleanup etc. (we save the journal to /var/log/upgrade.journal, and write /var/log/upgrade.log for those of you who don’t like the journal)

reboots back to your newly-upgraded system

Part of the goal was to make the process distro-neutral. Other than system-upgrade-fedora, none of this is really Fedora-specific. The upgrade framework itself (the dracut hooks and the systemd trickery) should apply to any distro using systemd and dracut.

The details are a bit vague here because the design isn’t finalized. The current (working prototype) design involves bind-mounts and doing systemctl switch-root twice, which has caused us a bunchofproblems. We’ve got some workarounds for this for Fedora 18 Beta but the design will likely change a little between now and Final.

I thought you’d like that! But please bear with me – I’m not bothering to get into the details of the existing implementation at the moment because it’s deeply flawed. The way it works now, there’s no obvious way to unmount the root partition. Data corruption ahoy!

So, yeah. At the moment we’re testing some awful workarounds for Beta, and I’m refactoring this bit to fix that. But I’ll be happy to discuss the details of the broken implementation after I’ve finished the mad scramble to fix it!

Does this also mean that the new upgrade system will use much less space in /boot? I have a “mature” Fedora install that still has the old 250MB /boot partition. The tricks needed to make pre-upgrade work are tough going (although I did at least document them when I last upgraded).

This is kind of a tricky question! There’s no inherent reason fedup wouldn’t work for upgrades of Fedora X to Fedora X+2, or X+3, etc. (Let’s call these “leap-upgrades” for short). So in theory: sure, that should work.

So far, in practice.. it doesn’t. At least not for F16. Everything builds and runs OK, but after you reboot to actually perform the upgrade, something gets confused during the switch-root back to the upgrade image, and the system hangs. F17 to F19 might work fine, but nobody’s tested that.

In general, though: supporting leap-upgrades is bound to be harder due to the larger differences between the stuff in the upgrade image and the stuff in the system. We don’t yet know how much harder, though. If it turns out to be a lot more work, it might not be officially “supported”.

But if it does turn out that way, that’s a policy decision, not a technical roadblock. There would very likely still be unofficial and unsupported ways to make leap-upgrades work, and I’d probably add code to facilitate that when possible. Especially if people send me patches.

Not really. In short: lots and lots of work, for (probably) not much gain.

First, we’d have to hack a bunch of stuff up to even be able to build deltarpms – the Fedora build systems don’t really deal with cross-release stuff at all right now.

Once that work was done, building and maintaining the deltas would mean using extra disk/CPU from the build system for every update to F17 or F18. This makes builds slower and uses a lot more space on the mirrors.

Also, the differences between F17 and F18 packages are a lot larger than the differences between two consecutive F17 updates, so the deltas will be much larger. This means taking up extra disk space on the user’s system and spending a lot of time to reconstruct the RPMs before the upgrade starts.

For users with sufficiently fast internet connections (or slow disks), using deltas would make things much slower – so we have to have some way of avoiding that. The current code gets this wrong for my systems all the time, and it’s only going to be worse when you’re dealing with thousands of bigger-than-normal deltas.

So. Definitely a lot of extra work for everyone involved, end users included. And we have no idea what the benefits would be.

If someone actually wants to sit down and test it out, I’d love to know exactly how much time it takes to build the deltas, and how big they are, and how long it takes to apply them, and how much bandwidth would be saved, and all that.

And if we get reliable data that shows that it’s actually a huge improvement – big enough to offset all the time and effort of implementing it – well, sure, that’s something we could probably do! But I suspect that’s not going to be the case.

Fedup should work fine with common RAID setups (e.g. /boot on RAID1). I asked the QA team to check that specifically, and nobody seems to have found any problems with it.

Considering the number of bug reports I’ve gotten so far, I’d be really surprised if RAID was busted and nobody bothered to tell me. If that happens: HEY RAID USERS, HOW COME YOU DON’T HELP TEST STUFF TO MAKE SURE RAID STUFF WORKS, HUH??

Many have ignored this, and set out in search of just the right combination of godawful hacks to fool RPM into letting them do it anyway. Most had their systems and sanity destroyed in the process. There Be Dragons.

One problem I always ran into with preupgrade, and seems even more serious (?) with fedup, is with how much free disk space I need. On several of my machines, I have root (not boot!) partition which contains /, and a separate /home partition. The root partition on one machine is 20 GB in size, with Fedora using 11 GB out of it.
Now, theoretically, I should have no problem upgrading it – the new release is only slightly larger, and definitely won’t need 20 GB.
But fedup appears to be very “generous” in the way it uses disk space – it loads all the RPMS into the root partition, which in my case is about 3 GB more. And then – what happens during the actual upgrade (dracut-fedup)? Do I need twice the space to hold two versions of each package, or just once?
All of this is very scary. It would have been nice if fedup did some calculations when beginning to run – whether the root partition is actually big enough for this upgrade.

It would have been nice if fedup did some calculations when beginning to run – whether the root partition is actually big enough for this upgrade.

It does, actually. Fedup runs an RPM transaction test after fetching all the packages, to see if RPM thinks it can install all the packages without running out of space or hitting fatal dependency problems. If you didn’t get any errors about that, then it should be fine.

If you like, you can download a copy of the DVD install image and use that as a data source for fedup, which will save you a bunch of space on the root partition.