One of my customers was concerned that someone or something was randomly deleting the /etc/niminfo on several of his AIX hosts. We discussed several ways in which he could track down who or what was removing this file. Obviously, the AIX audit subsystem would be an ideal method of capturing this event. However, PowerSC Real Time Compliance (RTC) can also monitor and capture this type of event.

“The PowerSC™ Real Time Compliance feature continuously monitors enabled AIX® systems to ensure that they are configured consistently and securely.

The PowerSC Real Time Compliance feature works with the PowerSC Compliance Automation and AIX Security Expert policies to provide notification when compliance violations occur or when a monitored file is changed. When the security configuration policy of a system is violated, the PowerSC Real Time Compliance feature sends an email or a text message to alert the system administrator.”

Here’s a quick start guide on configuring PowerSC RTC on AIX.

1. PowerSC is included with the AIX Enterprise Edition offering. Download the PowerSC software from IBM Entitled Systems Support (ESS) website.

The package name is ESD_-_PowerSC_Standard_Edition_v1.1.5_122016.tar.gz.

I like using the IBM Microcode Discovery Service (MDS) to perform spot checks on POWER system and adapter firmware levels. This tool gives me snapshot of a POWER systems current firmware levels. I can tell almost immediately if I need to update the firmware on the system or any device installed in the server. This report is especially useful when performing a health check on a customer’s system.

The tool is easy to use. First you need to download the latest inventory scout catalog file from the IBM website. The most current microcode catalog can be downloaded here:

Once downloaded, copy the catalog file to an AIX (or VIOS) partition on the physical server. In most cases, a Virtual I/O Server (VIOS) is the most logical place to copy this file. The VIOS typically owns all the physical adapter devices on the system.

# scp catalog.mic padmin@vio1:

padmin@vio1's password:

catalog.mic 100% 263KB 263.2KB/s 00:00

Next, login to the VIOS and move the new catalog file into the correct location for use with the inventory scout (invscout) tool.

# ssh padmin@vio1

padmin@vio1's password:

$ ls -ltr

total 968

drwxr-xr-x 3 padmin staff 256 May 17 21:20 tivoli

-rw-r--r-- 1 padmin staff 2389 Aug 19 04:43 smit.transaction

-rw-r--r-- 1 padmin staff 1290 Aug 19 04:43 smit.script

-rw-r--r-- 1 padmin staff 11247 Aug 27 11:57 smit.log

drwxr-xr-x 2 padmin staff 4096 Sep 15 15:26 SV810_081

-rw-r--r-- 1 root system 5103 Sep 15 15:53 install.log

drwxr-xr-x 2 padmin staff 256 Sep 15 15:53 fixes

-rw-r--r-- 1 padmin staff 176510 Sep 15 18:34 errlog.out

drwxrwxr-- 2 root staff 256 Sep 16 13:25 config

-rw-r----- 1 padmin staff 269478 Sep 17 20:18 catalog.mic

-rw-r--r-- 1 root staff 5143 Sep 17 20:18 ioscli.log

$

$ r oem

oem_setup_env

# ls -ltr

total 8

-rw-r--r-- 1 root system 88 May 17 20:26 catalog.mic

# cd /var/adm/invscout/microcode

# mv catalog.mic catalog.mic.old

# cp /home/padmin/catalog.mic .

# ls -ltr

total 536

-rw-r--r-- 1 root system 88 May 17 20:26 catalog.mic.old

-rw-r----- 1 root staff 269478 Sep 17 20:20 catalog.mic

#

Now all we need to do is run the invscout command to create a new MDS upload file (MUP file).

# invscout

****** Command ---- V2.2.0.20

****** Logic Database V2.2.0.2

Initializing ...

Identifying the system ...

Working ...

Getting system microcode level(s) ...

Scanning for device microcode level(s) ...

117 devices detected; each dot (.)

represents 10 devices processed:

...........

Writing Microcode Survey upload file ...

Microcode Survey complete

The output files can be found at:

Upload file: /var/adm/invscout/vio1.mup

Report file: /var/adm/invscout/invs.mrp

Report file: /var/adm/invscout/invs.mrrup

To transfer the invscout 'Upload file' for microcode

comparison, see your service provider's web page.

#

The new MUP file (vio1.mup) can be uploaded to the MDS tool at the following URL:

Immediately my eyes are drawn to the items highlighted in red. Both the PCIe2 4-Port (10GbE SFP+ & 1GbE RJ45) and PCIe3 RAID SAS Adapter have new firmware available for them. I should update the firmware on these devices! But what fixes are included in the new firmware? Well, directly below the devices report is the ‘Microcode by Type’ table.

I can simply click on the ‘Readme’ hyperlink in the report and a new browser window opens with the readme file for the adapters firmware.

So, should I install these fixes right away? How critical are they? Again, the MDS report provides you with a quick way to answer these questions. Under the ‘Severity’ column, you will see several terms that describe the severity of the related fix. If you click on the word ‘Severity’ you’ll be presented with an ‘Understanding the MDS Report’ window.

On this page you’ll find the definition and description of each severity type. This will make it easier for you to ascertain if a fix should be installed ASAP or if it can wait until the next maintenance window.

I often save the MDS report as a PDF and include it in my health check reports. Alternatively you can save the entire page as a HTML document and send this to the customer. This will ensure the hyperlinks still work and can be referenced by the customer as needed e.g.

By the way, if the microcode catalog file you used to generate the MDS report is not the latest, the MDS tool will warn you. If you see this message, simply download the latest file and send it over to the server where it’s needed and re-run invscout and upload the new MUP file e.g.

“WARNING: the microcode catalog used in surveying host vio1 is now out of date. Version 2014.09.10 was used for the survey and version 2014.09.17 is the most current. We recommend that you either download and install the latest microcode catalog, and run this program again, or run the MDS Applet and allow it to automatically update the catalog.”

A colleague was attempting to recreate a WPAR. The WPAR had previously
been removed using the WPAR Manager Web GUI. This worked, or so he tells me!

Anyway, when he tried to create the WPAR again from the GUI, using
the same WPAR name (wpar20), it
failed. The error claimed that /wpars/wpar20
already existed. But it didn’t!

Now, before I go any further, looking back at the issue now, it is
pretty obvious what the problem might have been. However, the solution was not
evident at first. This proves one thing for certain: some problems cannot be
resolved without TWO cups of coffee in the morning.

Of course, I resorted to the command line to see what was REALLY
happening.

The mkwpar command was
indeed complaining that /wpars/wpar20 already existed.

#
mkwpar -n wpar20

mkwpar:
0960-288 The /wpars/wpar20 file system already exists.

mkwpar:
0960-417 Specify -p to create this workload partition using the existing file
system data.

I could not find any directory or file system with that name,
anywhere under /wpars or elsewhere.

#
ls -l /wpars/wpar20

/wpars/wpar20
not found

#
ls -l /wpars

total
0

#

#
df –g | grep –i wpar

#

#
lsvg | lsvg –il | grep –i wpar

#

I ran the mkwpar with truss to try to find out where it was
finding the reference to /wpars/wpar20.
At this point you have probably already determined where the problem might be!

The truss output
pointed me in the direction of /etc/filesystems.....of
course! Another “facepalm“ moment!

#
truss mkwpar -n wpar20 > /tmp/wpar20 2>&1

#
vi /tmp/wpar220

statx("/wpars/wpar20",
0x30008AB8, 128, 010) = -1

kopen("/etc/filesystems", O_RDONLY|O_LARGEFILE) =
4

Whatever my learned colleague had done when he “removed” the WPAR
with the GUI, it left entries for the WPARs file systems in /etc/filesystems.