Starting from 2.10 onwards, Ganeti has support for parallely installed versions
and automated upgrades. The default configuration for 2.11 and higher already is
to install as a parallel version without changing the running version. If both
versions, the installed one and the one to upgrade to, are 2.10 or higher, the
actual switch of the live version can be carried out by the following command
on the master node.:

$ gnt-clusterupgrade--to2.11

This will carry out the steps described below in the section on upgrades from
2.1 and above. Downgrades to the previous minor version can be done in the same
way, specifiying the smaller version on the --to argument.

Note that gnt-clusterupgrade only manages the actual switch between
versions as described below on upgrades from 2.1 and above. It does not install
or remove any binaries. Having the new binaries installed is a prerequisite of
calling gnt-clusterupgrade (and the command will abort if the prerequisite
is not met). The old binaries can be used to downgrade back to the previous
version; once the system administrator decides that going back to the old
version is not needed any more, they can be removed. Addition and removal of
the Ganeti binaries should happen in the same way as for all other binaries on
your system.

When upgrading to 2.13, first apply the instructions of 2.11andabove. 2.13 comes with the new feature of enhanced SSH security
through individual SSH keys. This features needs to be enabled
after the upgrade by:

$ gnt-clusterrenew-crypto--new-ssh-keys--no-ssh-key-check

Note that new SSH keys are generated automatically without warning when
upgrading with gnt-clusterupgrade.

If you instructed Ganeti to not touch the SSH setup (by using the
--no-ssh-init option of gnt-clusterinit, the changes in the
handling of SSH keys will not affect your cluster.

If you want to be prompted for each newly created SSH key, leave out
the --no-ssh-key-check option in the command listed above.

Note that after a downgrade from 2.13 to 2.12, the individual SSH keys
will not get removed automatically. This can lead to reachability
errors under very specific circumstances (Issue 1008). In case you plan
on keeping 2.12 for a while and not upgrade to 2.13 again soon, we recommend
to replace all SSH key pairs of non-master nodes’ with the master node’s SSH
key pair.

When upgrading to 2.11, first apply the instructions of 2.11andabove. 2.11 comes with the new feature of enhanced RPC security
through client certificates. This features needs to be enabled after the
upgrade by:

$ gnt-clusterrenew-crypto--new-node-certificates

Note that new node certificates are generated automatically without
warning when upgrading with gnt-clusterupgrade.

Starting with Ganeti 2.0, upgrades between revisions (e.g. 2.1.0 to 2.1.1)
should not need manual intervention. As a safety measure, minor releases (e.g.
2.1.3 to 2.2.0) require the cfgupgrade command for changing the
configuration version. Below you find the steps necessary to upgrade between
minor releases.

To run commands on all nodes, the distributed shell (dsh) can be used, e.g.
dsh-M-F8-f/var/lib/ganeti/ssconf_online_nodesgnt-cluster--version.

Ensure no jobs are running (master node only):

$ gnt-joblist

Pause the watcher for an hour (master node only):

$ gnt-clusterwatcherpause1h

Stop all daemons on all nodes:

$ /etc/init.d/ganetistop

Backup old configuration (master node only):

$ tarczf/var/lib/ganeti-$(date+%FT%T).tar.gz-C/var/libganeti
(``/var/lib/ganeti`` can also contain exported instances, so make sure to
backup only files you are interested in. Use ``--exclude export`` for
example)

For going back between revisions (e.g. 2.1.1 to 2.1.0) no manual
intervention is required, as for upgrades.

Starting from version 2.8, cfgupgrade supports --downgrade
option to bring the configuration back to the previous stable version.
This is useful if you upgrade Ganeti and after some time you run into
problems with the new version. You can downgrade the configuration
without losing the changes made since the upgrade. Any feature not
supported by the old version will be removed from the configuration, of
course, but you get a warning about it. If there is any new feature and
you haven’t changed from its default value, you don’t have to worry
about it, as it will get the same value whenever you’ll upgrade again.

You may want to copy all the messages about features that have been
removed during the downgrade, in case you want to restore them when
upgrading again.

Install the old Ganeti version on all nodes

NB: in Ganeti 2.8, the cmdlib.py file was split into a series of files
contained in the cmdlib directory. If Ganeti is installed from sources
and not from a package, while downgrading Ganeti to a pre-2.8
version it is important to remember to remove the cmdlib directory
from the directory containing the Ganeti python files (which usually is
${PREFIX}/lib/python${VERSION}/dist-packages/ganeti).
A simpler upgrade/downgrade procedure will be made available in future
versions of Ganeti.

After running cfgupgrade, the client.pem and
ssconf_master_candidates_certs files need to be removed
from Ganeti’s data directory on all nodes. While this step is
not necessary for 2.10 to run cleanly, leaving them will cause
problems when upgrading again after the downgrade.

If you’re using Xen-HVM instances, please double-check the network
configuration (nic_type parameter) as the defaults might have changed:
2.0.4 adds any missing configuration items and depending on the version of the
software the cluster has been installed with, some new keys might have been
added.

No changes needed. Note that the drbd7-to-8 upgrade tool does a disk format
change for the DRBD metadata, so in theory this might be risky. It is
advised to have (good) backups before doing the upgrade.