Thursday Jan 16, 2014

When running dbnodeupdate.sh for updating a db server the "-l" argument is always required to specify the source location of the new release/update. From here we will now call this 'location' the repository'. The repository comes in two different flavors: as an ISO and as Oracle ULN channel. Let's start with the ULN channel.

Exadata ULN Channels

For the different Exadata releases starting 11.2.3.1.0 a corresponding 'base channel' is available on linux.oracle.com (ULN). Exadata customers can register to ULN with their CSI, subscribe to the channel (e.g. patch/release) they require and then synchronize one or more channels to with the local (in house) repository. This local repository can then be made available to the internal data-center by publishing it via a web server.

Note: it is not recommended to use an Exadata db server as local YUM repository

Instructions how to subscribe a system to ULN and synchronize to a local repository are the same as for Oracle Linux and Oracle VM, so generic instructions can be found on OTN here. The README of a specific Exadata release will always mention what channels are made available for that release. You can also find this information via MOS 888828.1 and also in 1473002.1.

Additional to the 'base channel', there is also a 'latest channel'. Currently for Exadata there is a latest channel for 11.2 and 12.1 releases. The content of the 'latest' channel will never remain the same (unlike the 'base' channel) as long as there will be updates for that 11.2 or 12.1 release. When for example a new Exadata 12 release will be published this will be added to the existing latest channel (in addition to a 'base channel' also made available). This is the primary reason for the 'latest' channel being much larger (and taking more time to synchonize) than a base channel.

For Exadata installations on release later than 11.2.3.2.1, the 'latest' channel brings additional options. With the latest channel it's now possible to specify what release you want to update to. For example when on 11.2.3.3.0 planning to update to a later (but not the latest 12.1.release, just as an example) you can use the 'latest' channel and specify the "-t" flag to specify what release you want to update to.

Note that this can only be done with the 'latest' channel and that without specifying the "-t" argument by default the db server will be updated to the most recent release it can find in that channel. Of course there is also the option to just grab the 'base' channel and update without specifying any "-t' option.

Examples

updating with a latest channel specifying no argument (latest release in the channel will be used) here

updating with the latest channel to a specific release that not exists (a typo) here

For those not able or willing to synchronize repositories with Oracle ULN there is also an ISO image available. The ISO image is built (and zipped) by Oracle and is only available for 'base' channel content. An ISO is ~1GB and about the same size as the sum of all packages of the corresponding base channel on ULN.

Using ISO or ULN

From an 'update' perspective there isn't much difference between using ISO or a http repository, only the location (-l) changes:

The ISO file should not be unzipped and there is no need to make an local 'loop mount' to use the iso - this is all done by the dbnodeupdate.sh script

Validations

For each type of repository some validation checks will be done to see if it a usable is a repository, checks are done for expected files and also if the available Exadata release in the repository is a later release than the one currently installed - because if not, an update would not be possible.

When specifying an http repository it's required to specify the top level directory containing the 'YUM metadata repository directory'. Typically this is the directory that has the 'repodir' directory in it. (see example here). When an http location cannot be identified as a valid repository an you would see a suggestion how to locate the right url.

In this post I will explain background and usage for both backups and how they integrate with dbnodeupdate.sh

dbserver_backup.sh

For backing-up and rolling-back Exadata dbserver OS updates the
dbserver_backup.sh script is used by dbnodeupdate.sh. For each upgrade
by default the dbserver_backup.sh script is executed. When executed (either manually or via dbnodeupdate), the
dbserver_backup.sh script creates a small snapshot of the 'active' sys
lvm. The active sys lvm is the primary lvm that your current OS image is running on. For example:

[root@mynode ~]# imageinfo

Kernel version: 2.6.39-400.126.1.el5uek #1 SMP Fri Sep 20 10:54:38 PDT 2013 x86_64
Image version: 11.2.3.3.0.131014.1
Image activated: 2014-01-13 13:20:52 -0700
Image status: success
System partition on device: /dev/mapper/VGExaDb-LVDbSys2In the above example the active lvm is /dev/mapper/VGExaDb-LVDbSys2.The snapshot is created to have a 'consistent' 'view' of the root
filesystem while the backup is made. After the snapshot is created, it's
mounted by the same script and then it's contents are copied over to the
inactive lvm. For lvm enabled systems, there are always 2 'sys' lvm's "VGExaDb-LVDbSys1" and
"VGExaDb-LVDbSys2". VGExaDb-LVDbSys2 will automatically be created (on
lvm enabled system) if not existing yet. For the example above, the 'inactive' lvm will be VGExaDb-LVDbSys1

Now, depending on how many files there are in the root (/) filesystem
(based on your active sys lvm) the backup times may vary. Previous Grid and Database home installation zip files in
/opt/oracle.SupportTools/onecommand will make the backup take longer (not
the restore, which I will explain why later). Same for those who have many small files (like mail messages in /var/spool) - the backup may take longer.

One of the first steps the dbnodeupdate.sh script will doing when executed is making a backup with this script. Now, if you want to shorten your downtime and make this backup before
the start of your 'planned maintenance window' you have 2 options: Either execute the dbserver_backup.sh script yourself or use
dbnodeupdate.sh with the "-b" flag to make a backup only before hand.

Example making a backup with dbnodeupdate.sh here (see 'Backup only' for 'Action')

When you then have the downtime for planned maintenance and already have
the backup you can then let dbnodeupdate skip the backup using the "-n"
flag.

Both Sys lvm's are 30GB each. The snapshot that will be created is ~1GB. It is recommended to keep this in mind when claiming the free space
in the volume group to make your /u01 filesystem as big as possible. (the script checks for 2 GB free space in the volume group)

Now, when the update proceeds, the current active lvm will remain the
active lvm. This is different than what happens on the cells where the
active lvm becomes inactive with an update. Typically you will only
switch active sys lvm's when a rollback needs to be done on a db server, for example,
an upgrade from 11.2.3.3.0 to 12.1.1.1.0 needs to be rolled-back. What
happens then is nothing more than 'switching' the filesystem label of
the sys lvm's, updating grub (the bootloader) and restoring the /boot
directory (backed up earlier also by dbnodeupdate.sh). Then, a next
boot will now have the previous inactive lvm as active lvm.

Rolling back with dbnodepdate.sh as in the example here (a rollback from 11.2.3.2.1 to 11.2.2.4.2)

After booting the node, it's recommended to run dbnodeupdate.sh again with the "-c" flag to relink the oracle home's again.

Remarks:

It's important, to make a new backup before attempting a new update.

In the above example, there is only talk about the sys lvm's. This
means custom partitions including /u01 are not backed up. For regular
node updates this is enough to rollback to a previous release but it's
recommended to also have a backup of other filesystems inline to your
requirements.

Nodes deployed without lvm will not have this option available

Rolling back db servers to previous Exadata releases with this procedure does not rollback the firmware

Backup / restore procedure owners guide chapter 7

The backup made with the procedure in chapter 7 of the Oracle
Exadatabase Database owners guide covers total node recovery. Like the
dbserver_backup.sh procedure a snapshot is used for a consistent view,
then in this scenario a copy is placed outside of the db server (via
NFS in this example). This procedure allow you to backup every
filesystem you require. In case of emergency - such as a non-bootable
system, the node can be booted with the diagnostic iso. For non-customized partitions an interactive script
will then question you to provide backup details and recover the node
completely. For customized partitions steps (which are almost the same) can also be found in the owners guide.

Advantages / Disadvantages

Both type of backups serve another goal. Also, these are just examples -
of course customized backup and restore scenario's are also possible.The procedure as described in the owners guide requires external
storage, while the dbserver_backup.sh script uses space on the node -
but that is also where the risk is. The backup made with dbserver_backup.sh works well for the purpose of
rolling back upgrades. With the automation of dbnodeupdate.sh rollbacks
can be done simple and quickly.

However - loss of critical partitions and/or filesystems will not be
covered with this type of backup - so you may want to
combine both types of OS backup. The general recommendation is
to use the default built-in backup procedure when running dbnodeupdate
to make easy rollback possible. But also backup the entire OS and
customized filesystems outside of the database server with an interval based on your own
requirements.

Tuesday Jan 14, 2014

In this and future posts I am planning to describe some new
functionality and background of dbnodeupdate.sh starting with Oracle Exadata Database Machine release
11.2.3.3.0. Some of this functionality will be directly available to the operator
via the interface and can actually be used via an argument, however, some of the recent changes are made to make patching even easier, reduce human error and downtime.

You may also find some of the 'new features' described in MOS 1553103.1
'Exadata Database Server Patching using the DB Node Update Utility'

Exclusion/Obsolete list

With updates to Exadata 11.2.3.3.0 or later some packages on the database server will
become obsolete. When updating a db server the dbnodeupdate.sh script will
mention an 'RPM exclusion list' and an 'RPM obsolete list' in it's
confirmation screen. The 'RPM obsolete list' will list the all packages
that will be removed by default during the update to 11.2.3.3.0 (or
later) when no action is taken by the operator.

If you would like to find out first what obsolete packages will be
removed you have to choose 'n' when prompted to 'Continue ? [Y/n]'. This
will stop your current patching session. Then look at the contents of the freshly created 'obsolete list' (it
should have the runid of your dbnodeupdate session in it's header). Example here

All the packages listed in the '/etc/exadata/yum/obsolete.lst' file will
be removed by default - this has multiple reasons, mainly this is because these packages are not
required anymore for Exadata functioning or they are considered a
security risk. In case you would like to keep for example the 'java'
package, you should create an 'exclusion file' which is '/etc/exadata/yum/exclusion.lst' and put the 'java*openjdk' rpm name (or
wildcard) in it.

Example:

[root@mynode u01]# cat /etc/exadata/yum/exclusion.lst
java-*-openjdk

When dbnodeupdate.sh is restarted you would see that the 'exclusion file' is detected (an example here).

All packages you have put in the 'exclusion file' will still be listed in
the obsolete file, but will not be removed when the confirmation screen says 'RPM exclusion list: In use (rpms listed in /etc/exadata/yum/exclusion.lst)' with the
update to 11.2.3.3.0 or later.

Frequent releases of dbnodeupdate.sh - keep an eye on it:

dbnodeupdates.sh and it's MOS note were designed / made to be released frequently and quickly when needed.This way dbnodeupdate.sh can provide workarounds for known
issues, emergency fixes, new features and best practices to the operator
relatively quick.This reduces risk of people not keeping up to date with 'known issues' or not being sure it applies to them. Basically the same idea as with the patchmgr plugins.

Also, unlike to the storage servers some customization can be done on the db servers - best practices and lessons learned from this in regards to patching may also benefit your environment. For those engineers involved with upgrading Exadata database
nodes, I'd like to emphasis to always check for the most recent release
of dbnodeupdate.sh when doing a db server update. I have already seen some people watching the releases closely, which is definitely a good thing.

About

Blog of Rene Kundersma, Consulting Member of Technical Staff at Oracle Development USA. I am designing and evaluating solutions and best practices around database MAA focused on Exadata. This involves HA, backup/recovery, migration and database consolidation and upgrades on Exadata.
Opinions are my own and not necessarily those of Oracle Corporation.
See http://www.oracle.com/technology/deploy/availability/htdocs/maa.htm.