Tips I Picked up at the Power Systems Technical University

In October, I attended and spoke at the Power Systems Technical University in New Orleans. While I was there I was able to attend some very valuable sessions and picked up some great tips. I thought this might be an opportunity to share some of those with you. Even though I have been working in the area for a very long time, I find these conferences extremely useful—both for networking and the technical content. At this conference I was focusing on several areas that I wanted to increase my knowledge on: enhanced mode HMC, networking and Spectrum Scale (formerly GPFS). Below are a series of tidbits I picked up. Hopefully you’ll find them useful as well.

Networks

One of the more interesting things I learned (from Alexander Paul) was very basic : how to quickly transfer an IP address without causing an outage. I’ve been moving my NIM server from fully virtualized to standalone and found this to be very useful.

In my case I was moving from en3 to en1

ifconfig en3 192.168.65.150 transfer en1
I then test with ping, etc and if it is working I make it permanent as follows:
chdev -l en1 -a netaddr=192.168.65.150 -a netmask=255.255.255.128 -a state=up
Be careful that you disable en3 before the next reboot or it
will come back up with the same IP address and you will have a duplicate
IP error. You can use chdev to set en3 to state=down or my personal favorite is to
rmdev the adapter and run cfgmgr so it comes back clean:
rmdev -dl et3
rmdev -dl en3
rmdev -dl ent3
cfgmr

I also went to a great session on Wireshark by Roy Spencer. I’ve been looking at a lot of trace output for VIO servers and the SEA has duplicate packets because the SEA sees the packet come in from the LPAR and the transmit out. It turns out that Wireshark includes a utility to remove these duplicate packets:

editcap -F nettle -d ‘source’ ‘dest’

This will help me a lot in my analysis especially when trying to produce packet counts.

Power Masters

The tricks of the Power Masters (Gareth Coates) session is always a great place to find nuggets. In particular, he provided details on how to turn on remote execution, remote operation and remote virtual terminals for the HMC in Enhanced mode. This is not under HMC management—instead you’ll find it under users and security, systems and console security. There’s a version of this session available at the Power Virtual User Group and a demo of the Enhanced HMC at the UK PowerVM Virtual User Group. I picked up a lot of tips on the HMC in this session including tips around RMC.

VIOS and Other Tips I Picked Up

export CLI_DEBUG=33 Then run a VIO command and it will show you the actual AIX commands that are executed behind the scenes.

As you can see, the SEA is on ent7 and the virtual ethernet carrying the traffic is ent4

Path Issues

Recently I have worked on multiple systems where the paths have been messed up for various reasons. It became necessary to remove the missing or failed paths without destroying the enabled paths. The way to do this is using the lspath command and get it to list all of the paths for that disk.

This is a great way to get the WWPNs in order to give them to the storage admins for zoning.

There’s also a -client flag on fcstat (as of VIO 2.2.2.2) that will get you information on statistics for all the WWNs and WWPNs that the VIO sees. You cqan run just fcstat -client to get full statistics or you can use the command to get the WWPNs and WWNs. You can also use the viostat -adapter command to get statistics. To get the WWNs and WWPNs, as padmin on vio2 I ran: fcstat -client | cut -c 1-50 Results were:

As you can see there are two excellent ways to get the same information, depending on where you want to request it from and what format you need it to be in.

Useful IBM Tools

I have mentioned these before, but there are a group of tools that I use all the time. They consist of nmon, nmon analyser, HMC Scanner, FLRT (fix level recommendation tool, FLRTVC (FLRT vulnerability checker) and fcstat -D.

nmon

nmon has been part of the operating system since AIX 6.1 and is used by admins all over the world to gather performance data. I like to run it using the following flags so that I don’t miss anything:

nmon -ft -AOPV^dMLW -s 15 -c 120

The above takes a 30 minute snapshot (120 x 15 second snaps) and includes asynchronous IO (A), the SEA (O) paging (P), volume groups (V), fibre adapter statistics (^), disk service times (d), memory pages (M), large pages (L) and workload manager statistics (W) if you are running workload manager. I run nmon all the time so my normal cron job runs for 24 hours using “-s 150 -c 576”. Depending on how granular you need to get you can tweak these values.

nmon analyser

nmon analyser goes hand in hand with nmon. It’s an Excel spreadsheet that processes nmon files and produces graphs of what is happening. It’s not a total performance monitoring solution, but it provides some valuable information for day to day performance work. I supplement this with my own data gathering scripts that gather more in depth data.

HMC Scanner

The HMC Scanner is a great tool for documenting your system in one place. The current version is 0.11.35—you actually download 0.11.24 and then replace the .jar file to get to 0.11.35. You can install it on AIX and run it from there against the HMC or you can run it from your desktop, assuming you have the correct Java installed. I normally run it from AIX using:

./hmcScanner.ksh servername hscroot -p password -stats

This produces a bunch of files, but the one you want is the one that ends .xls. This is the Excel spreadsheet with all the systems connected to the HMC documented.

FLRT and FLRTVC

FLRT (Fix level recommendation tool) has been around for several years now and it’s something I use whenever I am planning an upgrade or trying to determine how long I can stave off an upgrade for. The most recent version provides you with the release date for the versions you are running and their endo of service date (if announced). It also provides those for the recommended upgrade levels. It will also provide links to the readme files for the recommended updates.

FLRTVC is a vulnerability checker. You download the script and run it on the system to be reviewed. It uses wget or curl to try to download a file called apar.csv from IBM and it then checks known issues against your software levels. The most common things it finds are back levels of SSH, SSL and Java - by default these are at an insecure level and new levels need to be installed. At a minimum, Openssh should be at 7.1.102.1100 and openssl should be at 1.0.2.1100 to avoid security problems. Java should be at the August 2017 level which for Java7 is 7.0.0.610. I replace these on all my systems including my VIO servers. FLRTVC identifies the efixes and ifixes that need to go on, provides links to the readmes and also to the actual download. You can send the output to a .txt file, download it and open it in Excel (it’s in csv format) to make it easier to view. I typically run FLRTVC on my critical systems once a month.

fcstat

If you are looking at fibre adapter statistics then you will want to be familiar with the fcstat command, especially the fcstat -D. If you run fcstat -D against a fibre port it will provide the WWN/WWPN as well as the firmware version, the adapter speed that it is running:

Create a clone of a boot disk

The above copies the current rootvg disk to hdisk2 which will become the altinst_rootvg volume group. You can use this as a backup before making changes to the current system (simple reboot of the new volume group to recover), or you can then work on the new volume group using alt_disk_install and alt_rootvg_op commands. You can also make minor changes (such as adding a host to the /etc/hosts on the alternat disk) by waking up the volume group:

alt_disk_install -W hdisk2
You can now go and vi /alt_inst/etc/hosts

When you are done you put the volume group back to sleep:

alt_disk_install -S hdisk2

Whenever you use alternate disk install or any of its derivatives be sure to point the bootlist to whichever disk you want to boot from next. By default, many of the alternate disk install commands change the bootlist to point to the new volume group and this may not be what you want to do.

Summary

I found the Power Systems Technical University conference to be very valuable in terms of picking up tips to make my life easier as a sysadmin. Hopefully you will find these tips helpful as well. I highly recommend attending these conferences—there are some great sessions, but many tips get picked up during the networking discussions where you can compare notes with others. Many thanks to all those who presented and shared their ideas with myself and others.

IBM Systems Magazine is a trademark of International Business Machines Corporation. The editorial content of IBM Systems Magazine is placed on this website by MSP TechMedia under license from International Business Machines Corporation.

We use cookies to optimize your visit to our website. By visiting our website without changing your settings, you’re acknowledging your consent to receive cookies on our website. If you would like to change your cookie settings at any time please view our privacy policy for additional information. Agree