I like using the IBM Microcode Discovery Service (MDS) to perform spot checks on POWER system and adapter firmware levels. This tool gives me snapshot of a POWER systems current firmware levels. I can tell almost immediately if I need to update the firmware on the system or any device installed in the server. This report is especially useful when performing a health check on a customer’s system.

The tool is easy to use. First you need to download the latest inventory scout catalog file from the IBM website. The most current microcode catalog can be downloaded here:

Once downloaded, copy the catalog file to an AIX (or VIOS) partition on the physical server. In most cases, a Virtual I/O Server (VIOS) is the most logical place to copy this file. The VIOS typically owns all the physical adapter devices on the system.

# scp catalog.mic padmin@vio1:

padmin@vio1's password:

catalog.mic 100% 263KB 263.2KB/s 00:00

Next, login to the VIOS and move the new catalog file into the correct location for use with the inventory scout (invscout) tool.

# ssh padmin@vio1

padmin@vio1's password:

$ ls -ltr

total 968

drwxr-xr-x 3 padmin staff 256 May 17 21:20 tivoli

-rw-r--r-- 1 padmin staff 2389 Aug 19 04:43 smit.transaction

-rw-r--r-- 1 padmin staff 1290 Aug 19 04:43 smit.script

-rw-r--r-- 1 padmin staff 11247 Aug 27 11:57 smit.log

drwxr-xr-x 2 padmin staff 4096 Sep 15 15:26 SV810_081

-rw-r--r-- 1 root system 5103 Sep 15 15:53 install.log

drwxr-xr-x 2 padmin staff 256 Sep 15 15:53 fixes

-rw-r--r-- 1 padmin staff 176510 Sep 15 18:34 errlog.out

drwxrwxr-- 2 root staff 256 Sep 16 13:25 config

-rw-r----- 1 padmin staff 269478 Sep 17 20:18 catalog.mic

-rw-r--r-- 1 root staff 5143 Sep 17 20:18 ioscli.log

$

$ r oem

oem_setup_env

# ls -ltr

total 8

-rw-r--r-- 1 root system 88 May 17 20:26 catalog.mic

# cd /var/adm/invscout/microcode

# mv catalog.mic catalog.mic.old

# cp /home/padmin/catalog.mic .

# ls -ltr

total 536

-rw-r--r-- 1 root system 88 May 17 20:26 catalog.mic.old

-rw-r----- 1 root staff 269478 Sep 17 20:20 catalog.mic

#

Now all we need to do is run the invscout command to create a new MDS upload file (MUP file).

# invscout

****** Command ---- V2.2.0.20

****** Logic Database V2.2.0.2

Initializing ...

Identifying the system ...

Working ...

Getting system microcode level(s) ...

Scanning for device microcode level(s) ...

117 devices detected; each dot (.)

represents 10 devices processed:

...........

Writing Microcode Survey upload file ...

Microcode Survey complete

The output files can be found at:

Upload file: /var/adm/invscout/vio1.mup

Report file: /var/adm/invscout/invs.mrp

Report file: /var/adm/invscout/invs.mrrup

To transfer the invscout 'Upload file' for microcode

comparison, see your service provider's web page.

#

The new MUP file (vio1.mup) can be uploaded to the MDS tool at the following URL:

Immediately my eyes are drawn to the items highlighted in red. Both the PCIe2 4-Port (10GbE SFP+ & 1GbE RJ45) and PCIe3 RAID SAS Adapter have new firmware available for them. I should update the firmware on these devices! But what fixes are included in the new firmware? Well, directly below the devices report is the ‘Microcode by Type’ table.

I can simply click on the ‘Readme’ hyperlink in the report and a new browser window opens with the readme file for the adapters firmware.

So, should I install these fixes right away? How critical are they? Again, the MDS report provides you with a quick way to answer these questions. Under the ‘Severity’ column, you will see several terms that describe the severity of the related fix. If you click on the word ‘Severity’ you’ll be presented with an ‘Understanding the MDS Report’ window.

On this page you’ll find the definition and description of each severity type. This will make it easier for you to ascertain if a fix should be installed ASAP or if it can wait until the next maintenance window.

I often save the MDS report as a PDF and include it in my health check reports. Alternatively you can save the entire page as a HTML document and send this to the customer. This will ensure the hyperlinks still work and can be referenced by the customer as needed e.g.

By the way, if the microcode catalog file you used to generate the MDS report is not the latest, the MDS tool will warn you. If you see this message, simply download the latest file and send it over to the server where it’s needed and re-run invscout and upload the new MUP file e.g.

“WARNING: the microcode catalog used in surveying host vio1 is now out of date. Version 2014.09.10 was used for the survey and version 2014.09.17 is the most current. We recommend that you either download and install the latest microcode catalog, and run this program again, or run the MDS Applet and allow it to automatically update the catalog.”

This little tip was passed on to me by
a friendly IBM hardware engineer many years ago.

When entering a capacity on demand
(CoD) code into a Power system, you can tell how many processors and how much
memory will be activated, just by looking at the code you’ve given by IBM.

For example, the following codes, when
entered for the appropriate Power system, will enable 4 processors (POD) and
64GB of memory (MOD). I can also tell* that once the VET code is entered, this
system will be licensed for PowerVM Enterprise Edition (2C28).

IBM made some announcements today relating to
their latest POWER7 server offerings. The new line of systems includes new
entry level systems and the highly anticipated high-end system, the POWER7 795!
They also officially outlined some of
the new features available in AIX 7.1. You can review the details here.
I’ve discussed some of these new features here
and here.
The official AIX 7.1 announcement details are available here.

The announcement got me thinking about my
recent customer engagements and why some have chosen to deploy AIX into their
IBM POWER environments, while others are considering a Linux on POWER solution.

I’ve found that it usually comes down to a
skills decision more than anything else. Most customers are happy to either
continue working with AIX (if they are existing AIX users) or migrate from
another UNIX OS to AIX. I’ve seen very few customers actually migrate to Linux
on POWER, but I’ve worked with several that have seriously considered it. Those
that have chosen to deploy Linux are doing so purely because they have in-house
Linux skills. They are concerned that migrating to AIX may be too big a jump
for their technical staff. I find this thinking interesting, as most of the
customers I’ve dealt with who run other UNIX OS’s like Tru64, Solaris or HP-UX
are more than happy to migrate to AIX. They believe the move is relatively
minor and doesn’t require massive re-training of their UNIX admins. I tend to
agree.

For me, AIX
is my preferred “Enterprise class” UNIX Operating System. Notice I’m prefacing
this with the words Enterprise class. Don’t get me wrong, I have worked with Linux
systems in both small and large customer environments. It is a great OS. But
I’ve found that it really only fits into environments that have a relatively
small number of users and where significant downtime can be tolerated for
things like operating system maintenance. This doesn’t fit the Enterprise class
of UNIX server OS’s that I’m thinking of here. When I contemplate the word Enterprise, I think of servers and
operating systems that can respond to business demands in terms of performance,
reliability, stability and availability. An Enterprise UNIX can provide all of
these things without compromise. Linux can offer performance and reliability
(in my opinion). However, from what I’ve seen, it lacks features & functions
in the areas of stability and availability. AIX on the other hand ticks all the
boxes. Again, this is just my opinion based on my experiences with both AIX and
Linux in the Enterprise landscape. Others will no doubt have their own
experiences that may or may not match my own.

So when I’m designing an Enterprise UNIX
server environment for a customer, I always start with an AIX on POWER base. If
the customer wants Linux, sure I can look at that too, but I strongly recommend
AIX as my preferred choice for large systems. Most of my customers are running
relatively large SAP/Oracle systems. AIX on POWER is a great combination for
large Enterprise systems. If you need to deploy large database systems that
must service tens of thousands of users (like a big SAP system), then I believe
AIX is the perfect OS on which to provide a platform for these large scale
systems.

AIX is a very mature and powerful UNIX OS. It
has been a major player in the UNIX server market for over 20 years (as shown
below). Some people are just not aware of how mature, robust and stable the AIX
OS has become over the years. There are many impressive aspects of the OS in
the areas of performance, scalability, reliability, management and
administration.

Just looking at some of the administration
capabilities built into AIX are enough for me to always recommend AIX over
Linux (or any other UNIX OS), when it comes to large Enterprise servers.

For example, the System Management Interface Tool (SMIT) can make the UNIX admins life a lot simpler.
This is an interactive tool that is part of the AIX OS. Almost all tasks that
an AIX administrator may need to perform can be executed using this tool. It is
a text-based tool (there is also an X interface but I recommend sticking with
the text-based menus). Everything it does, it does through standard AIX
commands and Korn shell functions. This feature is especially useful when you
need to automate a repetitive task; you can have SMIT create the proper
command-line sequence, and you can then use those commands in your own script.
My compatriot, Anthony English, has a nice intro to SMIT on his AIX blog.

The
AIX Logical Volume Manager (LVM)
is built into the OS, for free. AIX LVM helps UNIX system administrators manage their
storage in a very flexible manner. LVM allows logical
volumes to span multiple physical volumes. Data on logical volumes appears to
be contiguous to the user, but might not be contiguous on the physical volume.
This allows file systems, paging space, and other logical volumes to be resized
or relocated, span multiple physical volumes, and have their contents
replicated for greater flexibility and availability. It provides capabilities
for mirroring data across disks, migrating data across disks & storage
subsystems, expand/shrink filesystems and more.......all of which can be
performed dynamically.....no downtime
required. The concept, implementation and interface to the AIX LVM is one of a
kind. All of its features support the ‘continuous availability’ philosophy.

One of the biggest reasons that I love AIX
over Linux is the mksysb. It’s built
into the OS and allows you to create a bootable image of your AIX system. This
image can be used to restore a broken AIX system or for cloning other systems.
The cloning feature is truly amazing. You can take an image created on a
low-end system and deploy it on any POWER system, all the way up to the
high-end POWER boxes. This simplifies the installation and cloning processing
when you need to install and manage many AIX LPARs. By using an SOE mksysb
image you can deploy consistent AIX images across your Enterprise POWER server
environment.

This brings me to another wonderful feature of
AIX, the Network Installation Manager
(NIM).
NIM is powerful network installation tool (comparable to Linux Kickstart).
Using NIM you can backup/restore, update and upgrade one or more AIX systems
either individually or simultaneously. This can all be achieved over a network
connection, removing the need for handling physical installation media forever.

Another fine example of AIXs superior OS management tools is multibos. This tool allows an AIX administrator to
create and maintain two separate, bootable instances of the AIX OS within the
same root volume group (rootvg). This second instance of rootvg is known as a
standby Base Operating System (BOS) and is an extremely handy tool for
performing AIX TL and Service Pack (SP) updates. Multibos lets you install,
update and customize a standby instance of the AIX OS without impacting the running and active production instance of the
AIX OS. This is valuable in environments with tight maintenance windows.

When it comes to upgrading the OS to a new
release of AIX, the nimadm
utility can assist the administrator greatly in this task. The nimadm
utility offers several advantages. For example, a system administrator can use nimadm to create a copy of a
NIM client's rootvg and migrate the disk to a newer version or release of AIX.
All of this can be done without
disruption to the client (there is no outage required to perform the
migration). After the migration is finished, the only downtime required will be
a single scheduled reboot of the system.

AIX
6.1 introduced new capability that most UNIX operating systems are still
working on. Concurrent
Updates of the AIX Kernel.....without a reboot! IBM is always working hard at making AIX an OS
that can provide continuous availability,
even if it needs to be patched. AIX now has the ability to update certain kernel
components and kernel extensions in place, without needing a system reboot. In
addition, concurrent updates can be removed from the system without needing a
reboot. Can you do this with other UNIX OSs?

And that’s just some of the features that make
AIX the only UNIX OS that I recommend
for Enterprise systems. There are many more tools and features that I couldn’t
live without (like alt_disk_install, savevg, installp, WPARs, etc, the list
goes on). If you are new to AIX and you are considering what your next UNIX OS
should be, then I recommend you take a very
close look at AIX.

And finally, the support provided by IBM, is
first class. Whenever I’ve needed assistance with an AIX issue or query, I have
always received timely, professional and useful advice. On the rare occasions
where I’ve uncovered a new bug, IBM AIX support have always been quick to
provide me with an interim fix to resolve or workaround a problem. That’s the
sort of support you’d expect for an Enterprise UNIX OS, isn’t it? What’s the
support like from your current UNIX (or Linux) OS vendor?

Linux is still a viable UNIX operating system.
However, I think it’s more suited to certain workloads like small to medium
size mail, web and other utility servers and services. AIX, however, would be
my platform of choice for my 10TB Oracle database running SAP ERP, not just for
performance reasons, but primarily because of the system administration
features of AIX that allow me to support and manage the system without
impacting my customers or enforcing reboots/outages whenever I need to change
something on the system.

All IBM need to do is create their own Linux
distribution (Blue Linux perhaps?)
that has all the features of AIX built in and then I’m sold! But why would
they? We already have AIX.