Share

"I'm My Own Grandpa" - Avoiding Dependency Hell

Written by James Weir - 15 october 2013

"I'm My Own Grandpa" is a song that tells the story of a man who, through a series of complicated marriages, becomes his own step-grandfather. Over the years, there have been many real examples that have been published in newspapers and magazines :

"There was a widow and her daughter-in-law, a man and her daughter-in-law, and man and his son. The widow married the son, and the daughter the old gentleman. The widow was therefore mother to her husband’s father, and consequently grandmother to her own husband. They had a son, to which she was a great-grandmother: now, as the son of a great-grandmother must be either a grandfather or great-uncle, this boy was one or the other" - Hood's magazine in 1884

Reading this reminded me of the confusion and pain that I have been through on the numerous occasions when I have had to install or upgrade packages on a system. A typical scenario would be, when installing a new package, I would quickly realise that this package required a whole bunch of other packages. Once those are installed, then guess what, I was missing their package dependencies as well, turning a simple installation into a nightmare. Welcome to dependency hell.

What is a Dependency

A dependency is a piece of information in a software package that describes which other packages it requires to function correctly. Many packages require the OS's system libraries as they provide common services that just about every program uses (filesystem, network, memory etc). For example, network applications typically depend on lower-level networking libraries provided by the operating system. The principle behind package dependencies is to share software, allowing software developers to write and maintain less code at a higher quality. Operating systems have thousands of packages. In the world of virtualization and cloud computing, it is becoming imperative to strip down the number of operating system packages to just the required packages to run a particular application. This process, known as JeOS (pronounced "juice") standing for "Just Enough Operating System" is a very painful manual process. So much so that many operating system vendors now supply a core operating system ISO with the minimum set of packages required to boot the system. The fun then begins as you manually install only the packages (and their dependencies) required to run your application.

Calculating Package Dependencies Automatically

One of the primary goals of the UForge platform was to avoid dependency hell when creating custom stacks (known as server or appliance templates). It was also important not to use core ISO images as the base of generating custom stacks as these are opaque, but to expose all the operating system packages in your template. Each appliance template has an os profile. This os profile contains all the operating system packages you have chosen. To help you get started, you can either choose from one of several standard os profiles (choosing the "bottom-up" approach) or use the smallest standard os profile available (choosing the "top-down" approach). You can then complement this package list by searching for other packages you require from the os distribution or upload and add any custom native packages (via "My Software").

So far you have not had to worry about package dependencies. You only need to add the packages you know you require to run your application (i.e. the first level dependencies your application requires).

Package dependency checking occurs when you generate a new image. During the first phase of generation, UForge calculates automatically all the dependencies of each package in the os profile as well as any packages contained elsewhere in your custom stack (custom software in "My Software" and "Projects"). All missing packages are automatically added to your os profile. For each package added, this package's dependencies are also checked. This process continues until all the dependencies have been met. The end result is a complete dependency tree of all the packages you require to run your application. What normally may take hours or days to do manually, only takes a few minutes.

Under The Hood

Each package has meta-data on what the package requires (that is, what the package depends on) and what it provides in terms of functionality. This meta-data varies on the package type (RPM, DEB etc). For example: RPM:

Provides - one or more libraries or services this package provides

Requires - one or more packages required for this package to run correctly

Conflicts - one or more packages this package conflicts with (the other packages cannot be installed if this package is installed)

Obsoletes - this package supersedes another (can be used for package renaming)

DEB:

Provides - one or more libraries or services this package provides

Requires - one or more packages required for this package to run correctly

Conflicts - one or more packages this package conflicts with (the other packages cannot be installed if this package is installed)

Obsoletes - this package supersedes another (can be used for package renaming)

Recommends - one or more packages that is recommended to be installed (optional)

Suggests - one or more packages that is recommended to be installed (optional)

UForge works with repositories. These repositories can be official operating system mirrors or private repositories. Other repositories including EPEL can also be handled by UForge. For each package in the repository, UForge uses the meta-data to calculate the dependencies in the os profile of your appliance template. The dependency calculation is done using a specific moment in time. This date is stored in your appliance (Appliance object, attribute: lastPkgUpdate). When creating a new appliance, the "lastPkgUpdate" date is the same date as the created time. Chosen package versions and dependencies are calculated by ensuring that they are equal to or less than the "lastPkgUpdate". Let's take an example. Imagine you create a new appliance on June 17th 2013, 17:00 GMT+1, and you choose package A, B and C in the os profile. Note that packages A, B and C may have more than one version (updates added to the repository due to bug fixing and or new features). The versions displayed for A, B and C will be dates of each of these packages closest (but inferior) to the "lastPkgUpdate".

In this case: package A v1.3, B 7.2, and C 3.2 is chosen. All dependencies for A, B and C will be calculated based on these versions using the package meta-data existing in UForge.

Wait A Minute, What About New Updates ?

As you probably know, packages evolve as bugs are fixed and new features are added. These new packages become available in the operating system repository. UForge uses an internal mechanism to check for any new package update available in the repository, and, if found, adds the meta-data of this package to its own database. Using this process, UForge builds a history of the operating system, as it keeps references to the old packages that are being replaced by the update. These updates do not get taken into account for your current appliance when generating a new image. UForge ensures the same package versions are used regardless of when you generate your image, the same image is generated time after time. This is due to always using the "lastPkgUpdate" timestamp of the appliance in question. Ok great, but what if I actually wanted to include these updates in my next generation? Well, it's a simple matter of updating the "lastPkgUpdate" of the appliance. UForge actually provides this update information for you. It knows the "lastPkgUpdate" of each of your appliances, as well as the list of packages in your os profile and therefore calculates whether or not any new updates are available.

In this case, UForge will notify you that three updates are available. Note, that for package B even if there is an intermediary package (version 7.3), only the last one is taken into account. UForge provides a way for you to see the evolution of package updates in the os profile. Clicking on the "update" button for a particular template produces a time graph of all the updates for the os package list through time. This allows you decide where to set the "lastPkgUpdate" to. By changing the "lastPkgUpdate" you can run a simulation that will describe which packages will be updated giving the current and new version of each package impacted.

By calculating package dependencies in this way, there is an interesting side-effect. Not only can you update (roll-forward) your packages, but you can generate an image from packages that were delivered in the past (roll-back). This is as simple as setting "lastPkgUpdate" to an earlier date. This is neat, as you don't necessarily need to always take the bleeding-edge latest updates of an operating system. This also goes a way forward to helping solve "destructive upgrades".

Destructive Upgrades

When you perform an upgrade of a package on a live system, the package manager will overwrite the files that are currently on your system with the files provided by the new package update. This works perfectly when you assume that the package you are replacing is backward-compatible. The problem is that it is extremely difficult to ensure this when delivering a new package version. Let's imagine that you wish to upgrade a package that depends on a newer version of php. When you upgrade, you will also upgrade the current version of php with the new one. If the php package is not completely backward-compatible then other applications that relied on the old php package may break. This is known as destructive upgrades. Once you have done a destructive upgrade, it is:

"...hard to undo, or roll back, an upgrade. Unless you or your package manager makes a backup of all the files that got replaced, you cannot easily undo an upgrade. Finally, while the package manager is busy overwriting all the files that belong to a package, your system is temporarily in an inconsistent state in which a package may or may not work properly. Hit the power switch on your computer halfway through your next OS upgrade and see if the system will still boot properly!" -- Pjotr Prins (Nix fixes dependency hell on all Linux distributions)

With the package "time machine", you can test whether you will have a destructive upgrade by simulating the upgrade, generating the image and testing it, prior to upgrading your production systems. Using UForge, you can also determine which packages will be updated and to which versions.

Sticky Fingers

Being able to roll-forward or roll-back the packages is all well and good, but what if we wanted to force a particular version of a package to be part of the generated image? Due to the current package version calculation being based on a particular date (the lastPkgUpdate) it is impossible to specify a particular package version to be part of the generation, as depending upon the build date of the package, potentially an earlier or more up to date version of the package may be chosen instead. To get around this issue, UForge provides a mechanism to enforce a particular package version. This is known as making a package "sticky".

In the os profile, by clicking on the "sticky" button, a list of all the versions of the package is displayed, allowing you to choose the version you require. Once chosen, this package becomes sticky. During image generation, this package version is chosen regardless of the current "lastPkgUpdate". All the package dependencies of this package are also calculated.

A simple click on the "sticky" package will make the package to become "unsticky", and the package version will be calculated as normal. To summarize, UForge provides powerful package dependency checking to create a JeOS appliance template with little to no manual effort - helping you to avoid dependency hell and ensure you will never be your own grandpa (or sister for that matter).